Fetch module

Initializes a safe TTL cache for storing multiple funding rates with a specified TTL.

Initializes a safe TTL cache for storing funding rates with a specified TTL.

Defines the time-to-live (TTL) for a funding rate as 5 seconds.

Defines an array representing possible numbers of orders to fetch.

Initializes a safe TTL cache for storing order book data with the default eviction TTL.

Defines the eviction time-to-live (TTL) for an order book as 5 minutes.

Initializes a LittleDict for storing order book functions keyed by order book level and exchange ID.

Defines the time-to-live (TTL) for an order book as 5 seconds (after which it is stale).

Defines the tuple type for OHLCV data, where each element represents a specific metric (Open, High, Low, Close, Volume).

Used to slide the since param forward when retrying fetching (in case the requested timestamp is too old).

Defines limit values for fetching futures data from exchanges.

Defines a NamedTuple structure for order book data.

This is the fastest (afaik) way to convert ccxt lists to dataframe friendly format.

convert(
    _::Type{Tuple{Vector{Dates.DateTime}, Vararg{Vector{Float64}, 5}}},
    py::Py
) -> Tuple{Vector{Dates.DateTime}, Vararg{Vector{Float64}}}

This function converts the provided Python object to a tuple format suitable for dataframes, specifically tailored for OHLCV data.

Propagates OHLCV data to all timeframes in a data structure.

propagate_ohlcv!(
    data::SortedDict,
    pair::AbstractString,
    exc::Exchange
) -> Union{Nothing, SortedDict{TimeFrames.TimeFrame, DataFrames.DataFrame}}

The propagate_ohlcv! function propagates OHLCV data for a given pair from an exchange exc to all timeframes in the data SortedDict data structure.

Converts a Python object to a DataFrame with OHLCV columns

to_ohlcv(py::Py)

Ensures dates are within valid range for the exchange and timeframe.

__ensure_dates(exc, tf, from, to) -> Tuple{Any, Any}

The __ensure_dates function checks if the dates from and to are within the valid range for the given exchange exc and timeframe tf. If the dates are not within the valid range, the function adjusts them accordingly.

Determines the starting date for fetching data.

__from_date_func(
    update,
    timeframe,
    from,
    to,
    zi,
    exc_name,
    reset
) -> Union{Returns{<:Tuple{Nothing, Any}}, Fetch.var"#19#20"}

The __from_date_func function determines the starting date from for fetching data based on various parameters. If update is true, it will fetch data from the latest date available. If reset is true, it will fetch data from the earliest date possible. The function also considers the timeframe, to date, timezone zi, and exchange name exc_name in its calculations.

Fetches OHLCV data for a specified exchange within a date range.

__get_ohlcv(
    exc,
    name,
    timeframe,
    from_date,
    to;
    out,
    cleanup,
    ohlcv_kind
) -> Tuple{Any, Any}

This function fetches OHLCV data for a given exchange exc, with the specified name and timeframe, within the date range specified by from_date and to. The fetched data is appended to the out data structure. The cleanup parameter determines if any post-processing should be done on the data before returning. The ohlcv_kind parameter determines the type of OHLCV data to fetch.

Determines the 'since' parameter for fetching data from an exchange.

__get_since(
    exc,
    fetch_func,
    pair,
    limit,
    from,
    out,
    is_df,
    converter
) -> Any

This function calculates the 'since' parameter based on the specified 'from' timestamp, or finds the appropriate 'since' value if 'from' is 0.0.

Handles errors during fetch operations.

__handle_error(
    e,
    fetch_func,
    pair,
    since,
    df,
    sleep_t,
    limit,
    converter,
    retry
) -> Any

This function takes an error e occurred during data fetching, and decides whether to retry the fetch_func based on the retry flag. If retry is true, it calls the fetch_func again with the same parameters.

Handles fetch operations for specified exchange and pair.

__handle_fetch(
    fetch_func,
    pair,
    since,
    limit,
    sleep_t,
    df,
    converter,
    retry,
    usetimeframe
) -> Tuple{Bool, Any}

This function calls the fetch_func for a given pair, starting from the since timestamp with a maximum limit of limit data points. It employs a delay sleep_t between fetches. The function also applies a given converter to the fetched data. If the retry flag is true, the function will try to fetch data again in case of an empty response. The usetimeframe flag indicates whether to use timeframe for fetching.

Returns an ordered list of timeframes for a given exchange

__ordered_timeframes(exc::Exchange) -> Tuple{Any, Any}

This function collects the timeframes from the exchange, converts them into periods, and sorts them in descending order. It then returns these sorted timeframes and periods.

Processes OHLCV data for a pair.

__pairdata!(
    zi,
    data,
    ohlcv,
    name,
    timeframe,
    z,
    exc_name,
    reset
) -> Data.PairData

The __pairdata! function processes the OHLCV data ohlcv for a pair name over a timeframe. It takes into account the timezone zi, data data, timezone offset z, exchange name exc_name, and a reset flag. If reset is true, it will reset the data for the pair before processing.

Cleans up fetched funding history data.

_cleanup_funding_history(
    df,
    name,
    half_tf,
    f_tf
) -> Union{DataFrames.DataFrame, DataFrames.GroupedDataFrame, DataFrames.SubDataFrame}

The _cleanup_funding_history function takes a DataFrame df of fetched funding history data for a name and performs cleanup operations on it. The half_tf and f_tf parameters are used in the cleanup process.

Iteratively fetches data over a specified date range.

_fetch_loop(
    fetch_func::Function,
    exc::Exchange,
    pair;
    from,
    to,
    sleep_t,
    out,
    converter,
    limit
)

This function calls the fetch_func function repeatedly until it has fetched data for the entire date range specified by from and to. Note: The total data points fetched may not match the expected number based on the date range.

Ensure a to date is set, before fetching.

_fetch_ohlcv_from_to(
    exc::Exchange,
    pair,
    timeframe;
    from,
    to,
    params,
    sleep_t,
    cleanup,
    out,
    ohlcv_kind
) -> Any

This function verifies that a 'to' date is set before attempting to fetch OHLCV data.

Fetches OHLCV data with delay for a given exchange and arguments.

_fetch_ohlcv_with_delay(
    exc::Exchange,
    args...;
    ohlcv_kind,
    kwargs...
) -> Any

This function fetches OHLCV data for a specified exchange exc and additional args. The type of OHLCV data to fetch is determined by ohlcv_kind. It applies a delay between fetches as specified in kwargs.

Wraps fetching function with error handling and backoff delay.

_fetch_with_delay(
    fetch_func::Function,
    pair;
    since,
    df,
    sleep_t,
    limit,
    converter,
    retry,
    usetimeframe
) -> Any

This function wraps a fetching function fetch_func with error handling and a backoff delay sleep_t. The fetch_func takes three parameters: pair, since, and limit, and returns a PyList. The converter function is used to tabulate the data such that the first column is the timestamp. The function will retry fetching in case of an error if retry is set to true.

Returns the name of an order book level.

_levelname(level) -> String

The _levelname function takes an order book level and returns its name.

Generates an order book of depth N.

_orderbook(
    N
) -> @NamedTuple{busy::Ref{Bool}, timestamp::Ref{Dates.DateTime}, asks::Vector{Tuple{Float64, Float64}}, bids::Vector{Tuple{Float64, Float64}}}

The _orderbook function generates an order book of depth N. The order book contains N levels of bid and ask prices along with their corresponding quantities.

Determines the start time for fetching data

_since_timestamp(
    actual::Dates.DateTime,
    p::Dates.Period
) -> Int64

This function calculates the timestamp from which to start fetching data. It ensures that the start time is not more than 20 years in the past or less than the given period.

Updates an order book in place with new data.

_update_orderbook!(exc, ob, sym, lvl, limit; init)

The _update_orderbook! function takes an exchange exc, an order book ob, a symbol sym, an order book level lvl, and a limit limit, and updates the order book in place with new data. If init is set, the function will initialize the order book before updating it.

Extracts futures data from a Python object.

extract_futures_data(data::Py) -> DataFrames.DataFrame

The extract_futures_data function takes futures data data from a Python object and extracts it into a format suitable for further processing or analysis.

Fetches candlestick data for a list of pairs from an exchange.

fetch_candles(
    exc::Exchange,
    timeframe::AbstractString,
    pairs::Union{AbstractString, AbstractSet, AbstractVector, Tuple{Vararg{T}} where T};
    from,
    to,
    ohlcv_kind
) -> Any

The fetch_candles function fetches candlestick data from a given exchange exc for a list of pairs over a specified timeframe. The from and to parameters define the date range for the fetched data. If from is not provided, it defaults to an empty string, which implies fetching data from the earliest available date. The type of candlestick data to fetch is determined by the ohlcv_kind parameter.

Defines the fetch limit for an exchange.

fetch_limit(
    exc::Exchange,
    limit::Union{Nothing, Int64}
) -> Union{Nothing, Int64}

This function fetches the limit for an exchange. If no limit is specified, it retrieves the default limit for the exchange.

Fetches OHLCV data from an exchange for a list of pairs.

fetch_ohlcv(
    exc::Exchange,
    timeframe::String,
    pairs::Union{AbstractSet{T}, AbstractVector{T}, Tuple{Vararg{T}}} where T;
    zi,
    from,
    to,
    update,
    reset,
    progress,
    ohlcv_kind
) -> Dict{String, Data.PairData}

This function fetches OHLCV data from a given exchange exc for a list of pairs over a specified timeframe. The from and to parameters can represent dates or, if from is a negative number, the function fetches the last N=from candles. If update is true, the function checks for cached data and only fetches missing candles. If reset is true, the function removes cached data before fetching. The progress parameter determines whether a progress bar is shown. The type of OHLCV data to fetch is defined by the ohlcv_kind parameter.

Prompts user for confirmation before fetching OHLCV data.

fetch_ohlcv(::Val{:ask}, args...; kwargs...) -> Any

This function prompts the user for confirmation before fetching OHLCV data for the specified arguments args and keyword arguments kwargs. If the user inputs 'Y', 'y', or simply presses Enter, it proceeds with the fetch_ohlcv function. If any other input is given, the function returns nothing.

Fetches OHLCV data for multiple exchanges on the same timeframe.

fetch_ohlcv(
    excs::Vector{Exchange},
    timeframe;
    account,
    sandbox,
    parallel,
    wait_task,
    kwargs...
) -> Union{Nothing, Task}

This function fetches OHLCV data for multiple exchanges over the same timeframe. It accepts:

• A vector of exchange instances excs.
• The desired timeframe timeframe.

The function can run in parallel if parallel is set to true. If wait_task is set to true, the function will wait for all tasks to complete before returning.

You can provide additional parameters using kwargs.

Returns the oldest possible timestamp for a pair.

find_since(exc::Exchange, pair) -> Any

This function iterates over the timeframes and periods of the exchange to find the oldest available timestamp for a given pair. If no data is found in any timeframe, it defaults to 1 day in the past.

Retrieves all or a subset of funding data for a symbol from an exchange.

funding_data(exc::Exchange, sym::AbstractString) -> Any

The funding_data function retrieves all funding data returned by an exchange exc for a symbol sym.

Fetches funding rate history from an exchange for a list of Derivative pairs.

funding_history(
    exc::Exchange,
    assets::Vector;
    from,
    to,
    params,
    sleep_t,
    limit,
    cleanup
) -> Dict

The funding_history function fetches funding rate history from a given exchange exc for a list of assets. The from and to parameters define the date range for which to fetch the funding rate history. Additional parameters can be specified through the params dictionary. The function will wait for sleep_t seconds between each request to the exchange. The limit parameter can be used to limit the amount of data fetched. If cleanup is set to true, the function will perform a cleanup on the fetched data before returning it.

Retrieves the funding rate for a symbol from an exchange.

funding_rate(exc::Exchange, s::AbstractString) -> Any

The funding_rate function retrieves the funding rate for a symbol s from an exchange exc.

Returns the appropriate OHLCV fetching function based on the specified kind.

ohlcv_func_bykind(exc, kind) -> Any

The ohlcv_func_bykind function determines and returns the appropriate OHLCV fetching function for the given exchange exc and kind.

Fetches an order book from an exchange for a symbol.

orderbook(exc, sym; limit, level) -> Any

The orderbook function fetches an order book from an exchange exc for a symbol sym. The limit parameter can be used to limit the depth of the order book. The level parameter specifies the level of the order book to fetch.

Parses a row of funding data from a Python object.

parse_funding_row(r::Py) -> Tuple{Int64, String, Float64}

The parse_funding_row function takes a row of funding data r from a Python object and parses it into a format suitable for further processing or analysis.

Updates the tail of an OHLCV DataFrame with the most recent candles.

update_ohlcv!(
    df::DataFrames.DataFrame,
    pair,
    exc,
    tf;
    ohlcv_kind,
    from
) -> DataFrames.DataFrame

The update_ohlcv! function updates the tail of an OHLCV DataFrame df with the most recent candles for a given pair from an exchange exc over a timeframe tf. The type of OHLCV data to update is determined by the ohlcv_kind parameter.