Control multiple bots with one master / controller / manager

[LoveIsGrief]

Describe your environment

• Operating system: OpenSuse
• Python Version: 3.6.12 (python -V)
• CCXT version: 1.55.2 (pip freeze | grep ccxt)
• Freqtrade Version: 2021.7 (freqtrade -V or docker-compose run --rm freqtrade -V for Freqtrade running in docker)

Describe the enhancement

Something like a freqctl to start, stop, list freq trade instances and control them programmatically / extensibly.
That means that freqctl could also be extended with strats of its own on when to start/stop bots, when to pause them, to manage their pair lists, rerun hyperopt in parallel every X period to tune the params of a strategy, etc.

Background

The ultimate goal is for the strategies to be chosen automatically depending on the configuration and performance. All of this should have an API (probably REST too) to have a UI for the manager.

Example: Strat1 trades live on pair_list_1 and is dry-run in parallel on pair_list_2. When the returns on certain pairs in pair_list_2 hit a lower water mark, they are moved to pair_list_1 and the corresponding bot is restarted.

---

I could contribute this myself either in this repo or as a separate project since there is a REST API, but I would like to get feedback first. Maybe someone has already written this and I haven't found it, but such a thing doesn't seem to exist according what I've gleaned from the docs.

P.S Thanks a lot for the bot! <3

[xmatthias]

to me, this sounds overengineered - at least most of it.
Deployment itself is a pretty complex topic - and pretty difficult to do right considering you can't run unlimited bots from one IP, but are limited to 2-3 (live, on binance, anyway) before hitting rate-limits.

With that - you'd need to go out to different hosts - which might have different systems (or different deployment modes - like native install or docker) - which makes it pretty complicated overall - and something that'll be VERY difficult to support.

---

My goal is to have freqUI support multiple bots from one UI instance (it's still quite some way to go - especially with time constraints).

Adding trading mode to freqtrade webserver is pretty simple (i've got a partially working branch with that) ... but there's some details that need to be fixed to improve stability (including long-time tests to see how stable it is when started that way and eventually fix/improve that).
freqtrade webserver on it's own also consumes almost no ressources, so you could have multiple up and running "on hold" until you need / want to start them.

Having the 2 above steps in place (UI supporting multiple bots, webserver supporting trade mode) - spinning up a new bot can be as easy as starting a new docker image with freqtrade webserver - adding it to your main UI instance (maybe https://frequi.netlify.app/ - but with a "nicer" url?), select a strategy and hit "go".

Due to time constraints, you can see both of the above as "long term" goals if i'm working alone to accomplish that - but with some help (you seem to be willing... ) - things can be sped up considerably.

The advantage would be to have everything combined under one umbrella, and the steps are really not THAT great, considering that 80% of the things are in place already.

---

Now for the "switching" logic - i'm not a huge fan of that, as that's basically a strategy on it's own, and is usually better done within one strategy (by combining different signals depending on market conditions).
You'll also usually not want to wait for a "low point" - but you'll want to "predict" the turn to another strategy before it turns sour.

If you're only talking about switching pairlists - then maybe #4493 can accomplish this (potentially in combination with the above).

[xmatthias]

As a quick update - the multi-bot feature of freqUI is "almost" working - after which i hope it'll only be a small step.

[LoveIsGrief]

I'm curious as to how that was implemented, since deployment should be a part of it. Is there a specific module I can look at? There doesn't seem to be a github project / roadmap issue tracking the progress.

We might be misunderstanding each other, because you interpreted my suggestion as over-engineered, but mention spinning up multiple bots in (separate?) docker containers, connecting them to a UI and controlling them from there. It seems to be just another client (yours web, my suggestion CLI) without the ability to start/stop the freqtrade processes. I'm guessing it will keep the processes running, but just stop the actual trading.

Users will probably have to control their freqtrade processes manually through the UI. This is probably what the result will look like architecturally

If I'm mistaken, then please correct me.

If I'm not, then my suggestion is adding the possibility to control the children:

• programmatically and that that be done in an extensible manner
• from the CLI

---

Now for the "switching" logic - i'm not a huge fan of that, as that's basically a strategy on it's own, and is usually better done within one strategy (by combining different signals depending on market conditions).

For clarity I'll differentiate between switching strategies aka switches and trading strategies aka traders. Is it possible to call strategies from other strategies in the current way the code is written?

Switches would need access to the performance of traders (avg. profit, median profit, sharpe ratio, pairs, etc.) in order to toggle traders on and off, switch out their pairs, make further changes to their configuration (dry run, parameters, and so on), backtest them with other pairs, run hyperopt, and whatever else the developer wants.

How would that work with the current IStrategy? Were all of that logic to end up in a single file with trading strategies ( for example RSI+BB, MACD + EMA, VWAP+EMA, yada yada yada), I fear it would make it very difficult to read and debug.

[xmatthias]

Freqtrade has multiple ways to run (docker is one, but native installation on windows, unix or macos are other options - each either within a virtual environment, or outside of that).
Having an official "manager" would have to support all of these options across multiple hosts (!) - so what you're asking for is in my opinion not a good idea, as the endresult will need to be overengineered to support different circumstances.

I'm talking about starting docker-images (or freqtrade processes) - so the user is in control of starting the process the way he sees fit, on the systems he sees fit (same system, multiple VPS hosts), with the installation method the user prefers / can use (not all systems can use all methods for one reason or another).

The user can use FreqUI to add the "empty bot process" to the UI, and start either backtesting, or trading in that bot.

[LoveIsGrief]

OK, I understand your concern now.

How do you feel about switchers (we can stick to that name or call them multi-bot strats, w/e you like)? It doesn't have to be in v1 and could be in v2 of multibot. Starting up the freqtrade processes could be for a v3 or later.

[xmatthias]

my concerns about process starting remains, and i don't see that as something i'd want to support.
Realistically, you can't run more than 2-3 bots on one host without running into ratelimiting trouble - so assuming you wanna support such a thing, it MUST be multi-host - which will be problematic (and there's other tools out there that can get that done, like ansible or similar).

The problem would however remain - what's the difference between "start a process and connect the UI to it" - vs. "create a host and connect the UI to the host" ?
the first, you can do via API (and it's already kindof possible assuming CORS is sorted).
The latter, you'll have to go to SSH (what if that remote host is windows? no SSH there ...). Knowing the freqtrade userbase, it's often people without any prior knowledge with these things - which means we'll end up as support for setting up SSH properly so this will work.
It's not that it can't work technically - and i'm sure i can set that up for myself - but i don't think it's possible to support as an open source project - not with this user-size, anyway.

---

Switchers - i'm not sure i understand - and i'm not sure how you imagine this to work.

However, the new freqUI version will have all existing API methods/data available for each connected bot (if necessary) - so i don't think data comparisons / aggregations will be a problem - but for the beginning, i'd focus on "combined dashboard" functionalities more than on providing cross-bot functionalities (that can be a long-term thing - but already getting a good cross-bot dashboard (and good UI/UX for that) will be a lengthy task i think.

[ErezAmihud]

It is not exactly the same but there is this repo that handles multiple instances of freqtrader:
https://github.com/Ph3nol/Trading-Bot