Ekeroot (00:02.511)
you can't do anything wrong.

Lars (00:04.782)
Hadratant Aran Oompapa It's the one that's supposed to be cam on

Ekeroot (00:06.267)
I'm going to find a button here. There maybe? Yeah. You can turn off the camera. Okay, thanks. Oh, that was where I was going to... Always very weird. Bass!

Lars (00:17.868)
Mm. I'll do a bass test as well, so we don't... Yeah, but that's pretty deep.

Ekeroot (00:24.076)
Right? It's not squeaky rot today. Very good.

Lars (00:29.678)
About GenService. Yeah. Do you want to start?

Ekeroot (00:34.552)
Yeah, come on. Yeah.

Ekeroot (00:38.859)
Yesterday I rewrote a

maybe not too important piece of code but the idea was the code took... Let me give you all the context. All the context. Yes, so Turing invented the Turing machine in the 30s and from there everything has come down.

Lars (00:55.621)
All of it.

Lars (01:06.284)
Widely considered a mistake.

Ekeroot (01:08.159)
Yes, we should have done that. Couldn't he just have broken codes like the rest of them? Okay, maybe not all the context, some of the context. So this system handles bookings and drivers basically. We can dispatch a booking to a driver or a job to a driver and if they don't answer in time...

they will get set as didn't answer in time state. But then we want to set them back to idle after 55 seconds for some reason. It's a very exact number. So this job had a...

quite advanced thing that checked which of the nodes were available and then picked a node, an available node and sent to that node run this task. And I think this was done before the Elixir task thing was made. So it was homegrown tasks and the task runner and all of that and everything was kind of...

It used a register to put things in and stuff like that. Quite, quite nice. But the downside of this was that it needed a bit of infrastructure that didn't really feel needed. And the infrastructure was a, a GenServer containing is this node up or not? So when you wanted to shut down everything, you said, this node isn't up.

Lars (02:48.852)
Huh.

Ekeroot (02:54.351)
And then it would start waiting for the tasks to be done. And when they were done, it shut off everything. Cool. So this, I ended up in this rabbit hole because I found or heard about that there's a functionality in Plug Cowboy to drain sessions. And...

Another thing that was homegrown here was functionality to drain sessions. And I thought we use plug cowboy anyway so why not do that and then we can remove this code. I love removing code. Uh, kinda. You know where the system grows horizontally. And this horizontal part is a GraphQL

Lars (03:24.657)
Uh-huh.

Lars (03:30.762)
You used to only use Cowboy, have you added plug?

Lars (03:36.278)
Kind of. Okay.

Ekeroot (03:48.867)
API and that uses plug. And then we have some other applications because Umbrella was fun for a while and they use the whole Phoenix stack. We have an Umbrella app. It's indeed it's this is the kind of things that happen when you start hiring developers. Suddenly you have resources to do stuff.

Lars (04:01.59)
You have an umbrella now?

Lars (04:05.706)
It didn't used to be an umbrella.

Lars (04:17.239)
Oh no.

Ekeroot (04:17.539)
Uh, and it could be good stuff. Could be bad stuff. I'm not judging it. It's just stuff. Um, yeah, absolutely. We've, we've learned so much. Um, things were learned. Knowledge was acquired. All the knowledge. So, uh, yeah. And my, my.

Lars (04:24.194)
things were done.

Passive voice.

Lars (04:32.034)
Things were learned.

Ekeroot (04:44.331)
I think one of my main things I do at work is to remove code. Because code that doesn't exist can't be detected. Right?

Lars (04:54.469)
Hmm.

Ekeroot (04:56.407)
And I don't like tectet. Because it slows us down. There are bigger fish to fry here. But gotta start somewhere. Yeah, this is a spicy fish. So...

Lars (05:08.002)
This one looked juicy.

Ekeroot (05:16.327)
When I had removed the part that led the session draining, it didn't work very well with our shutdown server thing in very exciting ways because our shutdown server thing shut down

Ekeroot (05:45.423)
there are no etch tables left containing the connections you want to wait for. Right? So then the draining stuff crashed. Yeah. But when that was fixed it kind of works. I haven't deployed it yet so we'll see. If you see lots of

Lars (05:53.835)
Yeah, no.

I could see that.

Ekeroot (06:11.363)
Small cute electrical vehicles burning. It's my fault Yeah, is the EV apocalypse Anyway My idea to I don't know if you have you have you kind of

Lars (06:17.25)
Got it.

Ekeroot (06:34.867)
Did the context travel from my brain to your brain? Do you understand what this whole thing is? Yes, it started there.

Lars (06:41.25)
So you wanted to drain sessions. And yeah, there was a thing for draining sessions, but you wanted to use another thing for draining sessions.

Ekeroot (06:50.427)
Yeah, and then because I want to remove the server module, because I think it's tautological in a way that doesn't really benefit us.

Lars (06:59.026)
Oh yeah, you want to not have a gen server that says this node is up, which I guess in this case means active. Like if it responds at all to that question, it is by definition up.

Ekeroot (07:09.592)
Yeah.

Ekeroot (07:15.771)
Yeah, but I can see that functionality which is like, okay, we go from full speed ahead to please do not send any more work to me. And then we can shut down. But I have a very strong feeling the beam handles this for me. I'm not sure. We'll see in the following weeks. Because it's like eye testing production and so do you.

So, back to the tasks. The tasks, this task is basically wait for 55 seconds, then set the driver we're waiting for to idle. But if we receive a message regarding this driver, stop the countdown. Just cancel.

Lars (08:11.143)
Yeah. So if he, if status actually changes.

Ekeroot (08:14.675)
Exactly. Because I think the driver can in their app say, press a yes button or something and then they get a new state anyway. The best thing with that code, which has been, I don't think it has been touched for many years, was that it was missing a hat.

So whenever any driver sent a message with the content state, it would cancel all count times. Yeah, best bug.

Lars (08:56.526)
idle resets.

Lars (09:01.181)
Ideal.

Ekeroot (09:02.855)
Yeah, so I fixed that first and now I just removed that code. So yeah, that's how it goes.

Lars (09:11.154)
I fixed this and then I made it so that it doesn't need to exist anymore. That's the two steps of fixing the bug.

Ekeroot (09:16.214)
Yes.

Ekeroot (09:19.992)
I think I need to rewrite a think piece containing about that on LinkedIn.

Lars (09:24.766)
Oh yeah, and then you can extrapolate this. So, and then you remove the feature at all, the entire feature. So maybe removing bookings in this case, or rides or drivers. Um, and then you can actually remove the code base and then the company. And now you've significantly simplified things.

Ekeroot (09:35.403)
Yeah, yeah, yeah. GRIVERS.

Ekeroot (09:46.673)
Yes.

Ekeroot (09:51.083)
Yes, I have become nothing the simplifier of things. I don't know. Something like that. Anyway, so I thought the real problem they wanted to solve was we need to keep these countdowns active regardless.

Lars (10:03.65)
Yeah.

Ekeroot (10:19.175)
of if this node is active or not. So I haven't tested this. So right now it works, right? I'm a positivist. So my idea of how to solve this is to have loads of send after timer things.

Lars (10:22.516)
Okay.

Ekeroot (10:44.815)
You get those from process, I think, the process module. And then you get, when you start one of those send afters, it gives you a reference, so you can read the timer, you can cancel the timer and so on. And the cool thing with canceling a timer is that you get back how many milliseconds is left. That's so cool.

Lars (10:48.279)
Yep.

Lars (11:08.263)
Mmm, I didn't know that.

Ekeroot (11:11.856)
I'm super hype about that.

Lars (11:15.166)
Well, yeah, and it's kind of useful in case it's like, oh, but it's actually just hit 0.001 essentially, and was gonna fire. Maybe you actually want to do something about that.

Ekeroot (11:28.319)
Yeah, all the raise condition goodness. So when this new GenServer receives a terminate signal, I catch it in the terminate callback and...

Lars (11:45.101)
Yep.

Ekeroot (11:48.863)
run cancel timer on all the timers that have been saved in the gen service state. And if there's another node in the cluster, I add the timeouts to that node in the exact same way that it was added to this node like in the normal operation thing. Yes! And the driver ID.

Lars (12:11.426)
by grabbing the milliseconds you get from cancelling.

Ekeroot (12:18.699)
And it's, so I run the ERPC call to the other server. And if there is no other server, okay, just do the thing. Put all the drivers in idle and stop.

Lars (12:28.126)
Yeah, because.

Ekeroot (12:35.903)
So I'm having fun with GenServers.

Lars (12:40.162)
Distributed systems.

Ekeroot (12:42.547)
Yeah, and since I haven't tested this, it's the best idea ever.

Lars (12:47.166)
Yeah, right now it's fantastic.

Ekeroot (12:49.4)
Yeah.

Lars (12:51.458)
Yeah, I have some fun with GenServer stuff going on at a client's where it's just like, there are some things we need to cache. Our servers are fairly stable, they don't shift around a bunch, I'll just write a cache that kind of optimistically writes on both nodes. We'll see.

how, at what point that stops being worthwhile. And if we often see the caches fall out of phase with each other, because that could be, in some of these cases, that would be bad, in some cases it doesn't really matter. But yeah, it's, you can do some stuff that's really optimistic, but there's also all these mechanisms for doing it.

Ekeroot (13:22.791)
Yep.

Lars (13:46.57)
in a fairly proper way, which is nice. It's a very nice nuance of canceling a timer that you get the remainder.

Ekeroot (13:57.131)
Yeah, so good API. So good.

Lars (14:02.242)
Can you just check the remainder on the timer as well? Yeah.

Ekeroot (14:06.903)
Yeah, read timer is the function. And if the timer doesn't exist, you get nil or something because you never know, you know.

Lars (14:15.435)
Yeah.

Ekeroot (14:17.931)
Yeah, I always get reminded of this table in Elixir in Action with replace all these systems with Erlang. And this is one of the things, because another way to handle this would be to... Could you put it in Redis or something and have a job that runs every second, five seconds?

Lars (14:25.942)
Yeah.

Lars (14:30.93)
Yep.

Lars (14:44.474)
Yeah, and it's like you could probably do something with Postgres. Like you'd definitely store timings in Postgres and like, but you would have a table that actively churns, which is not great. And for Redis, it's like, yeah, okay, that will work fine. Redis is great, but...

It's infrastructure that you probably don't need and you need to have a good mechanism for receiving triggers from it. And in the end, that will probably mean that you set up a gen server because that's how you hold onto a connection anyway. I helped the changelog folks. They didn't end up using the thing because they, they did some other stuff before they ended up integrating my, my approach, but

Ekeroot (15:20.204)
Yep, yep.

Lars (15:34.206)
to see if they could, because they were not running Erlang distributions and they weren't clustered. And they wanted to replace a cache module that was kind of old. And I took a stab at kind of, okay, but this is a way that you could do a very crude distributed cache and just use Postgres to

Ekeroot (15:40.158)
Okay.

Lars (16:01.294)
for coordination with the ListenNotify. And that was like a GenServer on each thing. And the GenServer was just responsible for keeping in touch with Postgres about the listens and the notifies.

Ekeroot (16:04.948)
Nice.

Lars (16:20.45)
and holding on to a reference for a next table.

Because GEN servers are great, but they're also terrible. If you model everything in your system with GEN servers, you better have a good reason, because they enforce total ordering. Like they're a bottleneck. They are supposed to be a bottleneck. They are a... That's where all the concurrency goes to die. Because...

Ekeroot (16:50.051)
Yeah.

Lars (16:52.642)
They can receive a bunch of messages, but they will only act on one at a time. You can do some stuff to make them less kind of blocking. For example, you can do async call responses where it's like you do a no reply on a call.

and then later you send a reply. But yeah, overall, like Gen servers are an intentional bottleneck. Like they capture messages and they act on them one at a time. They are one process, not many. And...

Ekeroot (17:34.202)
Indeed.

Lars (17:35.958)
That's not great for performance when you have something that's, for example, read heavy or write heavy, where it's like, oh, tons of stuff needs to happen. Ets is great at that. Like ETS is quite performant, like in memory storage that's built to be used and abused often for performance sensitive things. So what you do is you start

your gen server just to hold on to the reference for an ETS table, but you make it, I think it's public, I think the option is, which makes it accessible from other processes. And then you can add read concurrent and optionally or write concurrent, which means optimize this for read or write concurrency at the cost of more memory usage.

And then your functions for operating on this ETS table can still live in the GenServer module, but they are not calls or casts. They're just calls straight to the ETS table.

Ekeroot (18:46.434)
Wait.

Lars (18:48.298)
Yeah, so if you're dealing with ETS and you actually want performance, do not make the calls that access ETS go through the Genserver.

Ekeroot (19:01.215)
It's the side channel call thing.

Lars (19:04.394)
Yeah, like ETS is a solution for shared state in our language.

Ekeroot (19:08.679)
Nice. I haven't had the reason to use it as either. And I really want to say no here, but that's a lie. Yeah.

Lars (19:15.99)
Well, do you need to store anything in memory and operate on it quickly?

Lars (19:23.29)
You have so much stuff in memory, but most of it lives in like GenServer states and stuff. And if you do need it to only be operated on by one thing at a time, then a GenServer is the correct solution.

Ekeroot (19:37.803)
Yes, we do have a cache running on the Minesha, which uses etstables. So yeah, which is also a reason why we don't need more etstables. Well, I think we also use a register module from Elixir, which uses etstables. So, yeah.

Lars (19:43.794)
Yeah, yeah, that's also.

Lars (19:59.115)
I wonder if registry does use ads. Doesn't it just use pg? Maybe. I don't know.

Ekeroot (20:05.867)
I think it uses etch tables. It's something like three tables per registry depending on no three tables per partition and everything is you know how everything can be kind of kind of lots of details and complex and it depends see when it falls over into Erlang land that's registry but it's a very nice coating on the Erlang land

flex it is.

Lars (20:34.626)
Hmm. Yeah. Overall, I would say, well, um, yeah, I guess.

P, I don't recall now is PG actually cluster global or not. It is. Yeah. But PG has a, has a very straightforward interface as well.

Ekeroot (20:48.459)
Yeah, if I recall correctly. Yeah. So you go registry when you love your. Yeah. It's good stuff. Um, isn't yes. Is it PG that has a newer version? That's simply PG two.

Lars (21:00.206)
process groups.

Lars (21:09.01)
No, no, no. The original version was PG, but for as long as I've done Elixir and Erlang, I don't think PG even existed. There was only PG-2. Then they implemented a replacement for PG-2 called PG.

Ekeroot (21:13.72)
Yeah.

Ekeroot (21:23.231)
Hahaha

Ekeroot (21:28.873)
Oh, they're doing the doom naming thing.

Lars (21:31.675)
And that's the new one.

Ekeroot (21:34.591)
Yeah. Okay, so the next version will be called Ultimate PG.

Lars (21:40.514)
Now it will just be P and the G modulus separate.

Ekeroot (21:45.36)
Yeah, you have to combine them yourself.

Lars (21:50.264)
Yeah.

Ekeroot (21:50.851)
but it will be a really sweet, gentle soundtrack. So it's okay.

Lars (21:55.835)
I gave a talk at Öredev recently and part of that talk is just, it's like a lightning run through why is it so cool? Why should you use it? What can it do? Tons of stuff. But one of the parts... What?

Ekeroot (22:15.772)
Did you do any speedrun hacks?

Lars (22:17.846)
No, well, I kind of had to skip nerves for time. I really summarized nerves, but...

Ekeroot (22:22.227)
Oh no. Yeah. Okay.

Lars (22:28.282)
On the machine learning part, there's a bit where I mentioned that, like one of the challenging parts in machine learning.

is like orchestration because Python needs help with orchestration.

Ekeroot (22:46.446)
Yep.

Lars (22:48.454)
And usually they end up using something which is very similar to like the actor model as used by Erlang. Like libraries like Ray for example, supposedly kind of close to. I think they follow Robert Verding's first rule of programming. Any concurrent, any sufficiently complex concurrent system in another language.

Ekeroot (22:58.814)
Yep.

Lars (23:18.982)
It contains an informally specified bug grid and an incomplete version of half of Erlang. Something to that effect. But one interesting thing there is like, oh, so they built all this ML tooling in Elixir and now they want to distribute it across multiple nodes. Okay.

Ekeroot (23:32.993)
Lovely.

Lars (23:49.794)
500 lines of code. Done.

And while they were at it, I think they solved multi GPU as well. So distributing work across multiple GPUs on multiple nodes.

which feels, it all feels very Erlang and it was all, it's backed up by Erlang distribution, of course. And while they're, at the same time, they also released the batching support, which is how you get kind of good utilization out of machine learning, where it's like, oh, let's wait until we have a few requests.

for the same model so we can batch them into the GPUs. You have essentially a timeout and a cap on how many parallel generations you can run or inferences. It's kind of crude in some ways, but apparently this is something that's still kind of finicky to get done in Python. And it's like in Alexa, it's just like, yeah.

GenServers, hurling distribution, we got it. It's good, done.

Ekeroot (25:08.715)
Yeah, I think one of the great superpowers of Gen Servers is this, we only do one thing at the time. It's also one of the big cons of Gen Servers or downsides which you alluded to earlier. But it's a... yeah.

Lars (25:16.567)
Yeah.

Lars (25:28.382)
It's a potential foot gun because people think of them as something that provides concurrency. Well, what it really does is honestly remove concurrency and sure serialization and serial execution. But of course, if you have more of them and that's also one thing that I think you run.

Ekeroot (25:38.144)
And they do, but they don't.

Um, yeah, we have another little.

Yeah.

Lars (25:53.83)
into when you do Elixir, early on it's like, I'm gonna build a gen server for this. It's gonna take you about two minutes in production before you're like, I need more than one gen server. Or at least, sometimes it's like during the design phase, you'll run into this, or maybe it's after you've done a few of these singleton gen servers and have...

Ekeroot (25:59.778)
Mm-hmm.

Ekeroot (26:08.178)
Hahahaha

Lars (26:21.71)
tried to test them or where you're really upset at them. But it's like what you probably need, if you really do need to manage state, you probably need a dynamic supervisor and a gen server on that can be started multiple times on that. For one thing, any gen server that you can start multiples of is much easier to write tests for.

Uh, it is more flexible and it is more likely to not, uh, murder you in the future. But it's like, if I was dealing with, um, I was dealing with Google auth recently, um, it's fine. Um, and the goth library, but you have it as well. The Google auth library for Elixir. Um.

Ekeroot (26:58.36)
Yeah.

Ekeroot (27:06.931)
Oh no, I'm so sorry.

Ekeroot (27:14.563)
That's a good library though. Yeah, it's really good.

Lars (27:19.834)
Yeah, so if you only need one credential, then you can just start one instance of Goff, pass it like a JSON credential service account file or something and have it go off and do its thing. And whenever you need tokens, you just ask it for them. Now, if you are dealing with a service account and also the like user.

credentials of anyone who uses your app through OAuth and like you need to juggle multiple credentials. Suddenly you need a dynamic supervisor, which is like easy enough to do. The tool, the tools for it is good. Um, because you need to run multiple, multiple goths, you need a whole dance party of sad, dark people.

Um...

Ekeroot (28:16.589)
Yeah, preferably with polka music.

Lars (28:20.23)
Yeah, yeah, industrial dance to polka, is that what you're thinking? Yeah.

Ekeroot (28:23.199)
Makes it all much better. Yeah, it's so good.

Lars (28:29.514)
Yeah, because one of the things that golf will do for you is to keep track of token freshness and do the refresh token dance if need be and stuff. Yeah, really useful, really important. Also

But I, of course, want all of those or multiple of them to run or at least the ones that are currently in use. So anything that any user that rocks up and like starts their session, I need to pull their credentials, shove it into Gov, have it generate an access token from the refresh token, that kind of thing. So I can use it for all my API requests.

Ekeroot (29:15.002)
Mm-hmm.

Lars (29:19.222)
And I've had similar things where I've used gen servers for rate limiting. So we had some APIs where it's like, oh, we have like a dozen API keys for this. But we are only allowed to use these keys for these customers and these keys for these customers. Like there were a whole set of rules.

Lars (29:49.17)
Also, we can't make more than these many requests per minute. We can only make at most these many requests per hour. And it should not be more than this in a day, for example.

Ekeroot (30:05.411)
That's like, leap year levels of logic.

Lars (30:09.682)
Yeah, like the most important thing was just to have ways to, for one thing, avoid hitting the rate limit, like the stupid simple rate limits, and also have some control over the rate limits. And that was like, okay, per API key, we need, we will need

to keep track of how many requests we're making. This means that all requests should go through a single point to some extent. Like you could possibly do things where it's like, yeah, the request doesn't have to pass through. I don't remember exactly how I did it, but I know there was a dynamic supervisor involved because of course we only start the rate limiting manager once.

someone has actually requested something for a particular key.

Lars (31:14.134)
But building that kind of stuff out in most other ecosystems, I think, would end up being like, oh, counters in Redis. And yeah, that type of stuff.

Ekeroot (31:27.019)
So how many GEN servers do you need for this? Or what's the... Can you try to draw the infrastructure for me in that you ended up building?

Lars (31:38.579)
So I think in the end, it was like, essentially every customer needed a, had their own API key. And we were not allowed to, I did test and confirm that any API key would be good for any customer. So ideally, optimally, we would be able to just use the capacity offered by these API keys.

Ekeroot (31:49.112)
Yeah.

Lars (32:07.942)
and optimize them, but we were not allowed to by contract. So, yeah, so I had to do the boring thing where active customers could hit limits. But yeah, essentially it was just one dynamic supervisor and that dynamic supervisor went, where like there was that dynamic supervisor module had like a call thing.

Ekeroot (32:10.383)
Just pull them and go. Hahaha. Boo. Boring contract. Boo. Hahaha.

Lars (32:39.994)
where it would spin up a GenServer on one of the nodes and any attempt to use that key would go via that GenServer. And it would keep state to keep track of.

Ekeroot (32:58.491)
Cool, how did you?

Lars (33:01.854)
your relationship to the rate limits because of course there were no headers or anything telling you where you were. And we go to the limits. Of course we blasted away the limits if we did a new deployment, but this company deployed like every two weeks. So.

Ekeroot (33:10.564)
Yay.

Ekeroot (33:18.379)
Yeah, that's... That-

Lars (33:20.802)
And it was not sensitive if we accidentally hit the limits. It was more like we are consistently hitting these limits. And one of the behaviors, I think, with that was just like requests start failing.

Ekeroot (33:35.214)
Oh, okay.

Lars (33:35.466)
Yeah. So, um, but it, yeah, they, they wanted that under control because it was very noisy in the logs and also, um, it was very hard to find out why or what was failing and now I could go, Oh, this customer has hit this rate limit with details in the log messages. Yeah.

Ekeroot (33:39.089)
All right.

Ekeroot (33:51.247)
Yeah.

Ekeroot (33:56.537)
Ah, very good.

Lars (33:59.51)
You have done 2000 requests in the last minute and it is for this organization. It's like, okay.

Ekeroot (34:08.044)
Yeah.

Yeah, because I've been thinking of building stuff with dynamic supervisors and what I'm running into that might not even be a problem, but I think it is, is to find the children of the dynamic supervisor.

Lars (34:27.878)
Oh, but that's what PG is for, or registry.

Ekeroot (34:32.591)
Okay cool so you start a register 2 somewhere and then you can do the Vaya song and dance right?

Lars (34:39.984)
Yeah, so global registration and VIA will get you... So then you can get globally registered things under a very dynamic name like a string name. And that's kind of what you want for looking up.

Lars (34:58.87)
looking up that type of a more dynamic gen server. The reason you want them under a dynamic supervisor is just that like gen servers should be under supervisors, so they are under a supervision strategy. And dynamic supervisors are the ones that are intended for varying amounts of children, and ad hoc adding of children. So you have like dynamic supervisor dot start child, which is...

Ekeroot (35:22.808)
That's so good.

Lars (35:29.39)
thing.

Ekeroot (35:29.719)
Cool. If you ever feel the need to ad hoc add a child, use a dynamic supervisor. Yes, population control 2.0.

Lars (35:34.55)
Yeah. Yes.

Lars (35:44.588)
We're gonna rebrand mothers now to dynamic supervisors

Ekeroot (35:48.663)
Absolutely. Yeah. Yes. That's yeah.

Lars (36:02.782)
It is more gender neutral, so... I think it has potential.

Ekeroot (36:06.335)
Indeed, indeed. Yeah, we'll add it to the next batch of amazing marketing ideas.

Lars (36:13.39)
Yeah. Can houses be nodes or homes? Homes can be nodes, yes. And people that know each other are considered a cluster.

Ekeroot (36:18.731)
Absolutely.

Ekeroot (36:24.911)
Uh...

Ekeroot (36:28.607)
Yes, and children generally communicate by doing very loud message passing.

Lars (36:34.722)
Hmm, a lot of let it crash as well.

Ekeroot (36:39.251)
Yeah, fall on the face, restart.

Lars (36:42.978)
Yeah.

Ekeroot (36:45.687)
I'm all for this.

Lars (36:49.202)
Yeah. And something I appreciate with Elixir, this was also kind of in my talk as a part of my... So I grabbed my demo from ElixirConf, but I did it in five minutes instead of 30. Yes, that was a speed run of just like, oh, this is web UI driven by an Elixir server. This is machine learning inference. This is media streaming. Cool.

Ekeroot (37:05.765)
Speedrun, speedrun hacks.

Lars (37:20.118)
But what I kind of wrap up that demo with is just to say, and all of these use the same abstraction. And granted, like membrane, for example, for media handling feels a little bit different too. It has a little bit of a DSLE feel to it sometimes. And it's very particular. It is a odd.

Ekeroot (37:39.279)
Mm-hmm.

Lars (37:44.894)
implementation on top of GenServers, but it is fundamentally GenServers. It's just doing a bunch of things and it's doing pipelines of GenServers. So there's some nuance to how it differs from other things.

Ekeroot (38:03.54)
How do they get any kind of performance out of it?

Lars (38:07.436)
Um...

Ekeroot (38:09.375)
Magic or is the beam fast enough?

Lars (38:11.918)
So for much of it, it's just that the beam is fast enough. And when you're passing binaries from process to process, if they are larger than 64 byte, I believe it is, they are treated as big binaries, which are only copy on change.

Ekeroot (38:17.944)
Yeah.

Ekeroot (38:35.303)
Ah cool, I like big binaries and I cannot lie because copy on write is something something. Yes.

Lars (38:37.428)
Yeah.

Lars (38:41.342)
Yeah, so there are some optimizations that probably help them there as well. But yeah, but overall it's like, yeah, when you're using the NX machine learning stuff, what you are fundamentally setting up is like a gen server and you make some calls to it and you like a live view is just a gen server with some outward facing API surface.

which is also implemented just the same way that the inward facing one is, but you usually don't use atoms because that's bad in that context. Yeah. And similarly, like also with membrane, like fundamentally you're dealing with a gen server that you set up and then that one has some kids. So.

Ekeroot (39:20.727)
Yeah, because there's an upper limit.

Lars (39:39.766)
It's just all gen servers and whenever it's like, oh, I got some information here and I want it to be available elsewhere. It's like, Hmm, what approach could I take? Message passing almost always message passing. So it's like when I want to expose information from my pipeline to my live view, it's like, yeah, I'll just do Phoenix pub sub. And then I have kind of decoupled them.

Ekeroot (39:56.097)
Yay!

Lars (40:07.886)
Or if I really want them coupled, I can start the pipeline from the live view and I can pass in the live views PID and have that as a reference that messages can be sent to. But generally I kind of like to decouple it with a bit of PubSub because usually just a little bit cleaner. Yeah, yeah, yeah.

Ekeroot (40:18.904)
Yeah.

Ekeroot (40:36.043)
It's good until it isn't. We had, you know...

Lars (40:37.794)
Your entire system is decoupled with PubSub, but not Phoenix PubSub.

Ekeroot (40:41.779)
Yeah, so it's, well, they kind of copied Phoenix PubSub many years ago and it still looks almost the same. So we've checked the code and they are scarily alike. But one of the fascinating things when you build your whole system with this event passing style or whatever, or PubSub, is that you

You get a global state that just sneaks into everything all the time, which makes all actions you do can get funny and surprising side effects when you least expect it, which...

Lars (41:25.922)
Yeah, you're like with your system, it's like, oh, I want to write some data. I'm going to do a pubs.

Ekeroot (41:31.807)
Yeah, yeah, pubs are broadcast entity. And then it goes into cache database and to all places where things listen for things. We had a lovely bug on that theme some weeks ago or it's still there because we haven't figured out how to fix it yet, but, or it's not that high priority. So we have a very small cache in a gen server.

Lars (41:41.703)
Yeah.

Ekeroot (42:01.779)
that contains like the latest state on how the system would make routes and handle capacity. So we have a limited capacity because we can't just spawn new drivers and vehicles which is a shame.

So if there are too many bookings, we can't deliver them. And we have something automatic to handle this. Basically say no, do the responsible, what's the name for that one? Whatever. It's a good thing to have in all distributed systems anyway, to say no when you can't handle more load. Yes, exactly. So.

Lars (42:41.367)
Yeah, load shedding.

Ekeroot (42:48.439)
This Gen server listens for events from bookings, which is reasonable and good. To see, well when a new booking shows up or is deleted or whatever, we need to rerun the cache. Or we need to rehydrate it.

to make it not stale. And then it also listens to events from work shifts because we fetch work shifts from a third party system where all driver schedules are made. So when someone changes that and adds a schedule, adds a work shift or removes it or whatever, we need to know it because that changes the capacity. Cool. What's...

Even more cool is that to be absolutely sure that we have a clean state, that we've gotten everything we need from this third party system. Once every hour the data fetcher simply gets all work shifts that are in the future.

Otherwise it every minute or twice a minute only gets the one that have changed the last five or ten minutes. But once every hour it gets all of them. Each Workshift that's downloaded that's not in-state deleted or something like that. So you can just assume all Workshifts that are downloaded generate one of those broadcast entities.

And this cache system that does the capacity caching and so on, it listens for all work shift signals and for every one of those it starts a new job to update the capacity, does all batching stuff and all that great stuff. Which means that when you get 100 new work shifts the

Ekeroot (44:58.447)
database connection pool gets full and starts load shedding and starts screaming in the logs. Yeah, and that was one of those, yeah, this is a gift from above. So strange.

Lars (45:04.767)
Nice.

Lars (45:15.286)
But every one of those events triggers a new I should fetch updates.

Ekeroot (45:16.744)
Uh.

Ekeroot (45:25.562)
Yeah.

Lars (45:26.902)
But that also seems like a case where it's like, no, you probably only need one thing that fetches updates, right?

Ekeroot (45:35.179)
You only need the last one or rather you only need like one per time it takes to fetch updates. If you have if it takes three seconds to fetch updates you can ignore all the events between them and then... Yeah exactly!

Lars (45:54.444)
Like it's a it's a thundering herd problem

Ekeroot (45:59.179)
And then you read to run once more because things have happened between it. And the even more funny part here is that this is almost implemented in a way that works. A simple way that works. That's not perfect, but it works quite well for the booking signals.

Lars (46:00.064)
dog piling.

Ekeroot (46:22.047)
or booking events, but it's not implemented for the Workshift events because someone forgot it. And yeah.

Lars (46:32.834)
So when I was building that rate limiting thing, I had this case where it's like, oh, but what if someone fires off three quick requests and the gen server is not up? And the thing you can do then is you can look at Erlang's global module. I think this is one of the more appropriate approaches. I wouldn't swear to it.

Ekeroot (46:38.703)
Yeah.

Ekeroot (46:49.488)
Oh.

Ekeroot (47:03.672)
Mm-hmm.

Lars (47:03.99)
but there's a global.trans, which stands for transaction in this case, which essentially lets you do, oh, only do this if you can grab a global lock for this resource. And if you can't, then...

Ekeroot (47:10.167)
I've seen that one.

Yep.

Ekeroot (47:27.035)
I think I've used this one.

Lars (47:33.246)
There's a few things you can do, but you can essentially ask them to wait. So I've done a few different variants of this at different times, but fundamentally it's like, if I get someone that wants this, but I'm already fetching it, as long as I have that information, I can put them on hold. I put them in a receive block.

and then I make sure that they will get a message when it's done.

Which means, whichever is first of these hundreds of interested parties, I will... They will all get the same response.

Ekeroot (48:16.512)
Yeah.

Lars (48:21.718)
when it's done. But I will only fetch it once.

Ekeroot (48:29.562)
Yeah.

is the Michel de resistance solution to the problem.

Lars (48:35.374)
Yeah, and the global trans thing is only necessary if you need your request to be kind of globally unique. If the thing is mostly happening on one node, you can, I think you can do, well, a locally named process or something. There's a few ways of doing the thing.

But essentially making sure that only one gets to do it at once and the others get a message about it when it's over.

Ekeroot (49:16.483)
sounds very reasonable.

Lars (49:16.63)
Yeah, it's fun to work in an ecosystem that has these types of tools. Sometimes I wonder, like, how would I even do this in like Python? And for some of the cases, it's like you mostly can't or really shouldn't. Um, yeah. And then the answer is usually Redis.

Ekeroot (49:39.409)
It gets messy quite soon.

Yeah, and like you need to re-implement Erling yourself in a system that has the global interpreter lock.

Lars (49:48.039)
Yeah, you need more earline.

Lars (49:55.478)
Yeah.

Lars (50:00.371)
The Jill is a limitation. I think both the Ruby and Python are tackling strategies for trying to unwind it, but they need to be backwards compatible. So it's if.

Ekeroot (50:01.923)
way too exciting.

It's a harsh mistress.

Ekeroot (50:16.159)
Yeah, it's, I can't remember his name, but it's one of the guys behind Lofty. I think the name is Lofty. They are a Django consultancy in the US. They have a podcast called Friday afternoon deploy, which is interesting. I'm really fond of it, but it might not be to your taste.

They are very good at ranting. But one of the things the CEO has said that JavaScript tries to become less and less like asynchronous, while Python tries to become more and more asynchronous. And both are doing not a very good job of it, if I may say so myself. That's not his.

Lars (50:59.213)
Hmm.

Lars (51:06.19)
Yeah. That's something I really appreciate working on the beam is that I don't feel like I need to spend a bunch of energy to figure out async versus sync because it's almost, it is weirdly explicit. Like you can only do one thing at a time in a single process. It's linear.

And you have a bunch of tools to explicitly do other things. Like something I'm generally missing when trying to do async and await and stuff in JavaScript is just like the simplicity of the Erlang Elixir receive. It's like, no, just wait until it's done.

Ekeroot (51:54.509)
It's so good.

Lars (51:58.622)
And not like, oh, yeah, wait, and then you have a promise that you can like, not a weird await keyword that turns life into promises. It's like, no, no. Just, I just needed to wait, but I know that's absolutely impossible in JavaScript. Uh, apparently in.NET as well, I was like, hmm.

Ekeroot (51:59.096)
Yeah.

Ekeroot (52:19.533)
I'm out.

Lars (52:24.198)
I know.NET has a sync and a wait. So I was like, hmm, I had to poke some.NET code recently. I'm like, maybe I could do this and then this would make, and I was just in pain because it felt like JavaScript suddenly again. And I get that like async and wait might be kind of nicer abstractions. I...

Ekeroot (52:24.973)
Oh no.

Ekeroot (52:34.497)
Oh no.

Ekeroot (52:41.093)
Yeah.

Lars (52:51.538)
rather like promises, like I could deal with callbacks, but callbacks get unwieldy. So I really understand why people didn't love callbacks. Like they really run off to the right when you nest them, uh, and dealing with multiple ones that can go in parallel is a pain and promises kind of tackles that. Promises is a pretty good API for, for all this.

Ekeroot (53:18.827)
Yeah. Also, promises feels more like it belongs in JavaScript. When you start with async await, it's like, this is wait and if it's not, I cannot send around things.

Lars (53:30.986)
I get really annoyed when the system tells me that this function cannot be this way or you can't use this keyword here. It's like, I don't know what you want from me. And I get the same thing with like imports and requires and ECMAScript modules versus common JS modules versus whatever the other ones are called. It's like.

Ekeroot (53:39.832)
Yeah.

Ekeroot (53:55.672)
Yeah.

Lars (53:56.798)
I'm trying to use the new stuff and I get angry every time.

like arrow functions, I can probably deal with arrow functions.

Ekeroot (54:06.272)
I'm kind of fond of them.

Yeah, that's almost the level I'm at in JavaScript. It's... I'm feeling like such a Kermitian or Luddite or something when I'm trying to work in JavaScript. It's just... Yeah.

Lars (54:22.654)
I'm perfectly happy writing JavaScript when, so again, like Phoenix has pretty much hit the sweet spot for me because they ship ES build. And it's just like, oh, so I can write my JavaScript and I can use newer JavaScript in case I'm copy pasting something that someone else made or if there are some features I prefer to use. Yeah, yeah, the ES build can handle like arrow functions and all that.

Ekeroot (54:34.083)
You're so good. So good.

Lars (54:53.794)
But they're like, and I could use modules, but I usually don't need a bunch of modules when I'm doing like live view and stuff.

Ekeroot (54:55.011)
stuff.

Ekeroot (55:05.975)
Nah, you don't need the glue.

Lars (55:07.03)
Yeah, and it's like I can just write my small pieces of JavaScript and sometimes larger pieces of JavaScript, but I don't have to go all in on like, oh, but you're doing modules and you're doing webpack and you're doing NPM and stuff. It's like, I will vendor up like a few libraries without issue. Like, I don't, I don't mind. I've been considering building a small kind of vendering tool that can just steal things from NPM.

Ekeroot (55:30.095)
Yeah.

Lars (55:36.466)
through Elixir.

Lars (55:40.39)
I think I have a half working library that can actually just download the NPM stuff. The challenge is if it's bindings or has a build step, suddenly you're in trouble.

Ekeroot (55:41.652)
You need to...

Ekeroot (55:51.599)
Yeah.

Yeah.

Lars (55:56.022)
But that's also maybe the dependencies you want to avoid.

Ekeroot (56:03.86)
Indeed.

Lars (56:05.418)
I like my ecosystem. It's not that I don't like other ecosystems. Rust was fine. The poking I've done in Rust was fine. Elm is kind of weird, but I mostly think it's fine. I have my first PRs into an Elm project. Thank you, thank you. Oh, well, no, they're merged and I'm using them.

Ekeroot (56:25.703)
Oh, congratulations. Have they, have you got any feedback on it? Oh, cool. Cool. Yeah.

Lars (56:31.874)
This stuff doesn't work. It's trying to treat JSON as YAML and it's not working. Like JSON is valid YAML, but this thing was doing terrible escapes to basic strings. So it was no good. So I made it at least do a small heuristic. If this file ends with.json, just assume that you're gonna use the JSON parser. Don't even try the YAML parser. And...

Ekeroot (56:37.455)
Hahaha

Ekeroot (56:41.4)
Yep.

Ekeroot (56:48.619)
No. No can do.

Ekeroot (56:58.563)
Ha!

Lars (57:00.83)
Like, okay, if someone needs to be using YAML for their open API definitions, go have fun. But yeah, so this was an open API to Elm generator that was using for client work. And it didn't run. It didn't run on my thing. It ran on others.

Ekeroot (57:14.628)
Okay.

Oh, yeah.

Ekeroot (57:25.68)
Oh yeah, runs on their input.

Lars (57:27.318)
Yeah. But it's, it's a fun thing with kind of becoming a little bit more experienced, a little bit less afraid of things. It's just like, ah, this doesn't work. Huh. Why doesn't it work? Poke, poke, stab, poke, stab. Ah. Report issue. Poke, stab, poke, stab, because you can't assume that they'll be fixing it while, while you actually need it. So it's like, ah, fix it, vendor it, uh, send a PR.

Ekeroot (57:47.107)
Yeah.

Ekeroot (57:52.705)
Indeed.

Lars (58:00.778)
Yeah. No, I'm usually fairly communicative. I got a few PRs into Ash as well recently.

Ekeroot (58:01.393)
It's fixed. Please merge.

Ekeroot (58:11.411)
Oh nice. Yeah. I think the last PR I got merged was this language called Wiwa. Spelled Ju-I-Ju-A. And it's a...

Lars (58:19.648)
Okay.

Ekeroot (58:23.779)
You know when you want to do programming that's absolutely useless in any kind of commercial setting, but you want to do it for like hobby stuff, exploration, tickle the brain, all that good stuff. I use WeWa for that because it's an array language.

Lars (58:29.524)
Okay, yeah.

Ekeroot (58:45.495)
without variables that uses the stack heavily. You can hear that it's a quiet taste. And I love it. I'm not very good at it because it's, and I love that too. It's a complete super challenge to read and usually to write and to just, yeah, it's good.

Lars (58:51.92)
Ah, okay.

Ekeroot (59:12.723)
But they had one of those as a functional programmer. One of my hammers that I prefer to use is recursion. So whenever I can't use map or whatever, I go for recursion. And they had a recursion glyph in Weeba, which I used. And it didn't really work because I bottomed out or topped out the stack.

Lars (59:19.786)
Yeah.

Lars (59:38.01)
Oh.

Ekeroot (59:39.559)
So stack overflow, everyone is sad and so on. And then I learned from another programmer that you should use the repeat glyph instead and give it infinite repetitions and use the break glyph when you want to break. Good stuff. So I added a very small piece of documentation that says you probably don't wanna use the recursion glyph, use the repeat glyph instead.

the repeat glyph is on its way out because they created another glyph called it do. It's one of the other exciting things with this language is that it's very... things happen to it all the time. There's... it's not stable at all. So...

Lars (01:00:23.639)
Oh.

Lars (01:00:27.086)
That is exciting. It is not necessarily what you want all the time. But for that type of language, I'm sure it might be fine.

Ekeroot (01:00:34.029)
Indeed.

Yeah, because I only do Advent of Code and Project Euler in it. So it's, I don't expect my code to be working in a couple of months. And it's, that's cool.

Lars (01:00:39.854)
Okay.

Lars (01:00:50.794)
No, sounds like fun. Sounds like when I sat down and tried to do a, not quite a static site generator, I think I wanted to serve pages from memory in a C web server.

Ekeroot (01:00:53.85)
Yeah.

Ekeroot (01:01:08.155)
Oh

Lars (01:01:08.222)
And it's like, oh, I need to bring in like, okay, fair enough, I need to bring in a web server because I don't want to write one from scratch. That was easy enough to do. That's fairly reasonable. And then it was like, hmm, I do need to wrangle some JSON here because I have JSON in my...

Ekeroot (01:01:18.971)
Good start.

Yeah.

Ekeroot (01:01:32.442)
Yeah.

Lars (01:01:36.778)
in my pages that I have for my blog. So I was trying to actually be able to use the ones I already had. I was like, okay, maybe, okay, bring in a library for that. JSON is not like trivial to parse and see.

I started thinking about how I could do it, but that was like, no, I'm fine with using a library. So let's bring in a library. And that was actually kind of tricky to figure out how to use. And then, uh, I also brought in a Markdown library.

Ekeroot (01:01:59.772)
It's...

Ekeroot (01:02:05.06)
Yeah.

Ekeroot (01:02:09.986)
Yeah.

Lars (01:02:15.206)
Yeah, so I think I managed to convert some of my pages to it. And I think I managed to serve one of them. Like there were a lot of quirks that I needed to iron out. And I ran out of spare time and energy, but it was a really interesting challenge. Like, okay, how should I think about this? And if I want to do, if I want to do a really fast index for looking at pages, like what, what does that?

Ekeroot (01:02:15.999)
Now we're talking.

Lars (01:02:45.134)
mean? How do I structure that? Like maybe I want to do bee trees and stuff or like what do I what do I want to do here?

Ekeroot (01:02:56.798)
Prefix tree. Come on, it will be fun. I don't know if this is the right solution.

Lars (01:02:59.53)
And it was also stuff like, okay, but if I want to have multiple different types that are packed in a particular way in memory, like how do I do? It's like, what you're talking about is an array of structs. And potentially you might want to do a packed struct or whatever it was called. But it was like...

Ekeroot (01:03:23.128)
Yes.

Lars (01:03:28.246)
poking some C programmers in a programming community. I mean, I'm like, I'm so weak at this. I'm terrible at this, but it's also very, very fun. And it's not like C has the benefit of not being a very large language, which I think, I don't think I would get going if it was C++. And I'm also not at all attracted by C++.

Ekeroot (01:03:37.743)
Hahaha. Yeah.

Ekeroot (01:03:48.92)
Yes.

Ekeroot (01:04:01.565)
I can appreciate C++ because it's...

Ekeroot (01:04:09.055)
It's like walking around in a jungle. You can see all these systems interacting and it's... It's not a very friendly place to humans. But it's... And it's very, very loud.

and dark and moist but it's also very fascinating.

Lars (01:04:30.346)
Yeah, I haven't gotten to being fascinated with C++. But it's like one of the first, like this was many years ago now, but I remember trying to do something in particular with C and being absolutely fascinated with what is a, like logically, if you know your memory stuff, which I didn't particularly.

Ekeroot (01:04:47.995)
Mm-hmm.

Lars (01:04:59.018)
It makes sense, but you cannot return a string.

That's just so weird to someone who's been doing web development for eternity. You cannot return a string. It's like, well, you can absolutely get a string out of a function, but you have to provide a space for it. Like give it somewhere to put a string and then it will put a string there for you.

Ekeroot (01:05:05.157)
Yeah.

Yeah.

Ekeroot (01:05:13.199)
Yeah.

Ekeroot (01:05:23.937)
it

Ekeroot (01:05:27.663)
So one of the arguments to the functions is a pointer to a string, right? Yeah.

Lars (01:05:30.079)
Yeah. And then there's also like conventions for how to provide a buffer to a function. That was something I ran into as well, where it can be a pointer to a pointer or like a pointer to an array of pointers. I don't remember the details, but... And then there's another approach. And it's kind of like, where do you...

Ekeroot (01:05:42.829)
Yeah.

Lars (01:05:56.478)
reference and dereference. That's fundamentally the thing. But you have to have some kind of convention because otherwise stuff just blows up.

Ekeroot (01:06:00.911)
Yeah.

Lars (01:06:07.891)
Um, yeah.

Lars (01:06:13.114)
Ooh, I had an interesting Erlang and Elixir bug recently.

Ekeroot (01:06:19.943)
Go ahead, I want to hear all about them.

Lars (01:06:22.523)
So I was using the Pacmatic library, which is a very cool library by Evad Nevu. I might have mentioned it before. It does streaming zip archives. So you can hit a plug endpoint, so usually a Phoenix controller, but any plug endpoint will do. And it can trigger, well, any function really, but...

Ekeroot (01:06:35.215)
Wow.

Ekeroot (01:06:43.608)
Yep.

Lars (01:06:48.51)
If it produces a Pachmatic stream, and then you do like Pachmatic run chunked or whatever it's called, it will start spitting out chunks of zip archive based on the sources you told it to include in the stream. And those sources can also be streams, files on disk, in the memory data, like a mix. And if you, I think it might even be able to pull URLs if you...

Ekeroot (01:06:57.037)
Mm-hmm.

Lars (01:07:18.21)
give it like pre-signed S3 things and stuff. So it's very, it's very high powered. It can do a ton of things. You can also just pass it a function for what type of, well, a set of functions. That's like, ah, these will be generated when you ask for them, don't worry about it. So what you get is like the moment you click the export button,

Ekeroot (01:07:41.534)
Hahaha

Lars (01:07:47.326)
your archive download starts immediately. And then it goes as fast as the stream can process each file in sequence. It would benefit immensely from some look ahead because it can be starved depending on what your backing is. But essentially it works just really well. We had some problems with it though.

caused a memory explosion and then crashed instead of producing the complete archive. I didn't have this problem initially, but it was being flaky and then no one really tested it and now it was like, oh, now I want to use it. Oh, it blows up. OK, that's not great.

And what it turned out to be was that for most of the files were really, really small. So there were, and you can hook into the events of this library to produce, um, progress information or whatever, like, or logging or tracking metrics. Like you can do a ton with it, but we had one of those things for just updating the UI and sending our live view some information about what's

What's happening with this download? How many files have we processed? How big is it? How, yeah, that type of stuff. Which file am I on?

And it was like going through 7,000 files fine. And then boom, it would blow up. On a, well, it was 7,105 or something. It was quite arbitrary.

Ekeroot (01:09:27.195)
So 7001 is too much.

Ekeroot (01:09:33.703)
Okay, yeah, yeah.

Lars (01:09:36.09)
And what I realized when I was looking through it, it's like, oh, it actually chokes on a kind of larger archive.

Lars (01:09:46.558)
Why? And it would go from like 150 megs of total memory usage for the entire application to 14 gigs on my local machine. Yeah, and then something would die quietly, which is also kind of weird. I'm not, I'm still not sure why it died quietly. I think that it hits a particular limit and I'm surprised that doesn't

Ekeroot (01:10:00.379)
That's a lot.

Ekeroot (01:10:05.499)
Huh.

Lars (01:10:15.598)
cause an error, but we'll see. And when I dug into it, what it turned out to be is that this event update, like entry updated was the event in question, happened so many times during one of these larger files and so quickly because it was, so it happened every...

Ekeroot (01:10:19.063)
Yeah.

Lars (01:10:44.778)
X amount of bytes. And apparently, frequently enough, that I was overloading my live view with messages. So there were message buildups, and the messages had some data in it, apparently enough to explode to like 14 terabytes within moments.

Ekeroot (01:11:08.262)
Hahaha

Lars (01:11:08.806)
Um, Erling is pretty fast, yo.

Ekeroot (01:11:15.427)
This is such a distributed systems problem. It's so, so exciting.

Lars (01:11:17.615)
Yeah. And like the solution was just keep some information about time elapsed in these updates and don't send like you can update the states of this of the data you're holding that you will be sending to the live view, but don't send it all the time. Just send it every 200 milliseconds. That's fine.

And then it was like breezy, nothing, no issue at all. But it's a very cool library because it means that you can do exports that don't particularly disrupt anything else. You're just streaming the things through. You don't have to do the in-between store and zip, no heavy operations really. You're just pulling things from whatever backends you have.

Ekeroot (01:11:44.599)
Yeah, absolutely.

Ekeroot (01:12:12.311)
Yeah, just a few pointers in memory and the beam does all that.

Lars (01:12:16.467)
And the chunks of like, oh, we have a few kilobytes passing through here at any time.

Ekeroot (01:12:23.639)
Yeah, and the beam does all the worker scheduling and whatnot so you don't have to think about that.

Lars (01:12:29.618)
Yeah. So it's a bit, it's a very cool library, but, uh, be careful about how many messages you actually, uh, send out.

Ekeroot (01:12:35.483)
Peace.

Ekeroot (01:12:42.867)
one of the Swiss army chainsaws. Love it.

Lars (01:12:43.155)
Yup.

So I believe there's a limit on how large mailboxes are allowed to get, which is apparently much easier to hit in Erlang generally because the default behavior of a GenServer in Erlang is to leave messages that don't match a handle, call handle info, handle cost in the mailbox. So if you accidentally send a bunch of messages to a GenServer, you'll have buildup.

Ekeroot (01:12:53.614)
Yeah.

Ekeroot (01:13:07.32)
Yeah.

Ekeroot (01:13:11.737)
Yes.

Lars (01:13:16.174)
Um, which is not how it works in Elixir. You will have errors instead.

Ekeroot (01:13:17.253)
Yes.

Ekeroot (01:13:24.019)
Oh, that's much better. Let it crash.

Lars (01:13:29.79)
And that's where we stop.

Lars (01:13:35.488)
Endless.