Dream big on your own time

Amos 01:39

Right.

Chris 01:39

And it was also, it also took me like a year to get used to it and get good at it and figure out patterns for it.

Amos 01:49

So, and what about the people who live alone though? Like you have family.

Chris 01:52

Yeah.

Amos 01:53

Like the people who are just like sitting in their house alone.

Chris 01:56

I mean, I think, I think this is hard. I think it's, it's hard to do. It's hard to start working from home no matter what the circumstances are, right. And it's hard to do it under the best of circumstances. And when you, it's a thing that you chose to do, and this isn't, anything anybody's choosing to do right now, you know, all the people who have to do it right now and didn't spontaneously choose to do it. They like need to do it. They have to do it. And that's a huge, that's, that's wildly different and it's, and you know, it's hard to do under the best of circumstances. And this is obviously not the best of circumstances. So it's really, uh, yeah. So I, I was out as, uh, I had, uh, an embarrassing first reaction, which I will now share because this is how we grow as human beings. Um, and now, uh, to everybody out there who's having a hard time, like, I, I, I empathize. I feel, yeah. I feel, yeah, it's, it's tough right now. Even under the, even under, yeah. It's like, you're a programmer and you can easily do your work from like, wherever you sit with a computer and uh Slack and the internet connection, you know, it's hard, it's hard.

Everybody's going through a lot, lot of stress and anxiety, you know, even, again, under the best of circumstances I deal with, I'm dealing with constant anxiety, which is.

Amos 03:13

On yourself?

Chris 03:14

I think just anxiety, you know, my anxiety levels have grown over the past basically four years, just sort of continually and the culminating and, and sort of, uh, in this where it's like, I I've never really had problems with actual true anxiety and the way that people have actual like general anxiety.

Amos 03:34

Right.

Chris 03:35

And, uh, yeah, no, I'm just, you know, just, uh, it's definitely a very difficult I've, uh, you know, having to, uh, be very, very, very selective about when I consume news, how I consume news, you know, how I consume certain conversations, walking away and from conversations, that sort of stuff, knowing when to ignore things.

Amos 04:02

I normally don't read a whole lot of the news, but I find myself reading news every day, Read a, read a white paper yesterday from Imperial college, uh, on the coronavirus study and how long they kind of expect this to last, uh, looking at all the data so far and some modeling is, is pretty crazy. But I, I would say that my reaction was similar to yours at first. Cause it was like, okay, I've been working from home or like, uh, an office by myself for a long time. Like this is something I can do, but it was more like about like me being able to survive. And then the longer I sit around, I'm like, oh, I'm very extroverted. We have no more user groups. We have no conferences. Like none of all of that stuff has stopped right now. So I mean, and then, then immediately it was like, okay, well maybe we can do our user groups remote, like we're just use Google Meet.

Chris 04:57

Right. Yeah. Figure it out other ways.Yeah. Figure out other ways to do it. Zoom calls or whatever. Yeah.

Amos 05:02

Yeah. So I'm, I'm excited to actually see that work too. I know that whenever we can get out and start gathering in groups again, I'm going to be excited because that's just who I am like, oh yes, I get to go be around people. But in the meantime, I think it's going to be interesting to see how, how we like as a society, um, and, and in our career field go from a lot of companies still pushing for, we have to have colocated teams to through the course of this, they're going to learn how to deal, hopefully, learn how to deal with people who are not colocated. And then is that going to change the way that our society looks coming out of this? Are we going to start having most of these big buildings full of people kind of fade away? Cause that's a big cost for companies. And if they realize at the end of this, oh my gosh, all this cost just went away.

Chris 05:59

Yeah. I, I that's, it's, it's super interesting. I don't know what's going to happen with that. You have to imagine that some number of companies we'll see, we'll start to realize, start to have that, that realization. Especially a lot of companies who'd never, would never have allowed this prior. So you got to imagine that that's going to shake up a lot of the industry. I think we're probably pretty safe to say that companies will, companies will become way more remote, friendly, they'll just, out of necessity. Like this isn't going to go away anytime soon, right. So we're going to, we're looking at, you know, yeah. I mean, there's all sorts of estimates around it. Some of them are highly depressing, but, um,

Amos 06:37

Yeah.

Chris 06:38

But, uh, and probably accurate. The depressing ones is the, probably the accurate ones, but perhaps.

Amos 06:43

How long this goes?

Chris 06:44

Yeah. But-

Amos 06:45

Right now I've read four months to 24 months.

Chris 06:48

Yeah. I mean, I think it's probably going to be close. I'm going to take the over on that one.

Amos 06:53

Yeah, I think so, too.

Chris 06:54

But anyway, all that said without getting, I don't want to get too deep in the weeds and just make this, uh, so topical, um, about, about the virus and everything. Uh, because I do think we should provide, maybe let's, let's call it a little bit of an escape from that, but I think that's, uh, yeah, it's all, it's almost certainly that that big companies are going to become slightly more remote friendly, uh, especially ones that you would never have considered that. And we'll see. And then some may just crave going back to the way things were after it's, after people are free to do so. But, um, I have to imagine some of them at least keep those, those things in place.

Amos 07:30

And yeah, I think, I think at least some of the workforce isn't, is going to say, hey, I really, I maybe, I really enjoyed being at home. This is what I want to do.

Chris 07:39

Some workers just aren't ever going to want to do that. I think, I think it's, it's hard enough that some people are just not gonna be interested in that and just want to be around people. People are, you know, some people are just social and that's fine. That's all good. I think even those of us who've been doing remote work for a long time. There's a funny phenomenon that happens. So BR engineering, especially backend, is probably almost all remote, right. From a percentage standpoint, I think, I think back in engineers we're probably like, I'm just doing the math in my head and we're probably 80, at least that's, that's a low, that's a low amount.

Amos 08:19

Percent remote?

Chris 08:20

Yeah. 80% remote. I think the percentage is higher. Everyone we've hired recently has been remote. And so we're, we're very, you know, we were meeting, manager and I were talking about this last night, like how just lucky we are , privileged, like all of it, right? Like we live in a pretty rural area. So we have a ton, we have a big yard, we have hiking trails around here to go get- so the kids have plenty of ways to like get out of the house without, you know, while still being isolated. Uh, we have a great job. Well, I already work from home, you know, uh, all these benefits and like the company's like already remote, basically remote, highly remote friendly, if not basically remote first. Um, and yeah, it's just, it's been pretty straightforward for us to transition into. It's been a lot harder for the people who did go to the office and, uh, who are used to that sort of, who are used to that. And so a lot of Slack channels are very noisy these days. Went from nothing to highly like just like the signal to noise ratio, flipped upside down, like, I mean, people are just, they're just craving that sort of, um, they're craving that, that communication and connectedness still.

Amos 09:43

I do know that, uh, a lot of developers that I've, that I talk to on a pretty regular basis that go into the office, reached out to me and they're like, hey, where do I hang out with people online? Like, what do I do? You work remote? How do I do this? And um, so I just, I, you know, community wise, I just try to point them at like the Elixir Slack is at least a good place to start to talk to some people. Um, and then they're, I know find some groups that have maybe a smaller contingency of people, especially as Slack gets more noisy, like you were saying. So find some, a smaller set of developers that you're close to, or that you want to talk to from that you met at the last conference and just start your own Slack or Discord or whatever. Those are, those seem to be like the suggestions that people are like, oh yeah. And it makes them feel like they can do something about it.

Chris 10:40

Yeah, I agree. Yeah. And just, yeah. Finding just those ways to connect. This is hard, you know, and, but yeah, doing those, uh, they're, they're useful. They make you, I don't know. I don't have anything novel to say about this. Um, you know, you feel connected to people. That's good. Yeah.

Amos 11:00

So I have been talking to companies too, about just how to go remote and communication is the biggest thing. There's, there's so much whenever you're not remote that happens with with body language and context of like where you're standing or what you're looking at. And whenever you are in a chat, you know, you say "this "and they're like, what's "this "or it. And so there's like a lot of relearning how to, how to communicate really well, I guess. And I think at the end of this, even if it only goes on for a short period of time that there we're, hopefully we'll be a lot better at communicating.

Chris 11:43

Yeah. I hope so. I don't know. I certainly hope so. I think people are going to have to adapt one way or the other and you know, one of the benefits, one of the things that you do, if you are going all to an office, you know, that sort of necessarily means they're all in the same time zone, that makes, that makes communication a lot easier. Cause you can just do a Zoom call. One of the hard things that I've learned is, uh, all the ways in which I typically fall back on communication, uh, don't work on a truly remote team. Like Zoom is not sustainable or, you know, uh, physical meetings are not sustainable, even remote, on, with a large enough team and uh, and distributed over enough times zones.

Amos 12:29

Right.

Chris 12:29

Right. You only get a handful of times zones. Uh, you only get a handful of hours in the day where you're all online, that aren't either over someone's lunch or whatever. And so you have to move a lot of that discussion into something asynchronous and there- and you just lose a ton of fidelity when you do that. Like the truth is, and I, and I'm pro remote work. I'm also pro fixing these things. You lose a ton of fidelity when you go remote, a hundred percent, just a ton of fidelity. Like you lose a ton of signal, not being in the same room as someone else because you're right. You have all this body language, you have all this stuff and you've just got, you know, a human, just, you just pick up on stuff like humans have an innate sense of that, right. Or, you know, most of them do, not everybody, not everybody's good at picking it up social cues, but most people are pretty good at picking up subtle social cues and they're often unsaid. And when you're on a, the minute you go out of the same room, you lose a ton of fidelity. So the, you know, you lose a ton of the ability to read that stuff on a call.

And then when you go from a call to some sort of written thing, even if it's in like a pretty well-formed, you know, GitHub discussion or email thread or whatever, you lose way more fidelity than if you were on the call with each, because you just losing all like tone of voice, intonation, you're losing easily, laughter you're losing, you know, you're, you're missing out on all those things that you just pick up on, uh, when you're talking to somebody.

Amos 14:14

Sarcasm is really terrible does not come off.

Chris 14:17

Yeah, doesn't translate, doesn't translate.

Amos 14:19

It comes up as you being a complete jerk.

Chris 14:21

Basically. Yeah. And so you lose that. And then like if you go from some sort of long form well-constructed format to something crappy like Slack or IRC or any, any like very quick hit, you know, chat program, those suck, like I hate Slack's whole marketing thing of like, we got rid of all your emails, like screw you , like, email was so much better because it actually allowed people form real thoughts. Like -

Amos 14:53

Yeah, I think people in Slack, so there's a lot-

Chris 14:57

Slack is not communication. Let's be clear about that. Slack is not communication, no chat, and I'm using Slack as like the colloquial I'm using it the same way, I'd talk about Kleenex, right. Or Frisbee, right. It's just the brand name for the thing.

Amos 15:11

But, but like the big problem there is, you know, you get, you can even have two or three conversations in Slack at the same time with the same person. I see it happen a lot where like somebody asks a question while somebody's typing the answer, they ask another question. That's like, the first one was about work, the second one's about family. And then a third one's about like, what are you doing this weekend? And they get answers, get intermingled in there. And then pronouns start coming out instead of nouns. And it just gets confusing to follow. And, and I, you know, that, I like in some ways that if you have something like that, if you actually sit and wait for the other person, instead of trying to have multiple conversations like you would in real life, you can't like, while they're getting ready to answer, you, can't just like jump in and ask them another question, well you can, but you terrible that intermingling. When people wait, I find that communication and chat works really well because there can be a back and forth. It's a lot easier than email and you get things, I think sometimes slim down a little more than the giant emails that I used to get.

But if people aren't waiting, waiting, and they're using a lot of pronouns, then it gets hard to follow anything.

Chris 16:25

But I think there's also, there's the timeliness of it, right? So if you're in a reasonably popular channel and you're not on like, there's this perception that you then need to be online all the time. Um, and if you're not on, and again, if you get getting back to the time zones thing, right? If you just happen to miss the important conversation where you decided on what the Kafka payload was going to be, and it's in Slack somewhere, that's not visibility like that, wasn't a discussion.

Amos 16:50

And then you're in trouble because you didn't know.

Chris 16:53

Right. Or, you know, whatever. And so it's like you have to move to higher fidelity things when possible, but finding ways to do that, that are asynchronous that allow people to get involved based on their time zone. Like that's really hard. It's really challenging. And I don't think there's a really good solution for it. I think it's a lot of effort. And so I do think that there's, it's, it's an interesting problem space because I like remote work a lot, but I think this is a very unsolved problem, like a highly unsolved problem.

Amos 17:24

So what were the hard things, when you, you said it took you a year to transition, what were those things that hurt? And maybe like, like, it sounds like some of them are going to be Slack related, chat related. For me, it was like all communication. And I loved being around people all day. And then suddenly I was around like physically around nobody.

Chris 17:44

I think that's part of it. I think the, the physical isolation from other people was hard. I think communication was definitely, is definitely still a thing I struggle with and I'm working on and you know, I feel like I'm getting better at that, but it's, it's a long, long process to get good at that kind of this sort of communication. And that was a big part of it. I think the really the really, um, draining things, the things that were really the hardest to learn was how to disconnect from work. Like when your work is at your house and your computer is at your house and you're at your house, you know, you really have to start thinking about modes that you're working in, like modes of your life. And you're either in work mode or you're in home and life mode. And you really have to get good at walking away from both of those things. Because at the end of the day, like you need to have a life still. Even if you're just sitting there by yourself, you need to not work all the time. That's just good for you. You need to sleep. You need to not be staying up late, just working. You need to do all these good things for your life.

Amos 18:58

One of the things I've seen work for some people- your office, actually, in your house, would work great for this. Cause you have a door outside and a door that goes inside- is that to go from your, the rest of your house, to your office or your office to the rest of your house, you actually walk outside and around.

Chris 19:13

Oh, right, sure. Yeah.

Amos 19:14

Um, just to give you like that mental thing of I'm leaving home. Cause the biggest thing for me, well, so it was different when I worked remote, but I had an office that was like three miles from my house. It was just, I was alone in a room with no windows and nobody else in the building worked with me and it was mostly empty buildings. So that's how I first went remote. Um, cause I lived in the middle of the woods and didn't have internet that was even good enough to chat with. So, but then when I, when I moved to Kansas City last year, I started working out of my basement. And the hardest thing for me was like, I would run upstairs to grab a glass of water or something. And then I would be like, oh, there's like five dishes here. I'm just going to wash these real quick. Uh, oh, uh, let me stop by and talk to my wife. Oh, my daughter wants a hug. And by the time I didn't realize that I went to get a glass of water, which should take me two minutes, maybe, maybe five and I'm gone for a half hour.

Chris 20:12

Right, right. Right.

Amos 20:13

And then because of that, I also feel bad for clients. So then instead of working an extra half hour at the end of the day, I work an extra four hours a day.

Chris 20:22

Right. Well, and that's the flip side of it, which is that you also have to work still. You have to get work done still, you know, your, your job, you get the benefit of being at the house. That is a huge benefit. Um, it's really hard for awhile. I would never go back to not working from home if I could help it at this point, but you still have to do your job. Like the privilege you get for working at home is that you have a job to do. And I think that that's reasonable. Um, and so it means that you have to have good boundaries about when you're working too. And that was actually the, you know, a lot of it, a lot of the challenging bit was , me and the, the kids and my wife all getting on the same page about what that meant. Um, and we have a really good strat- we have like a really good pattern or a bunch of really good patterns with it now, um, that allows me to be involved in the kids' lives throughout the day, without having to, you know, always sort of disrupt the work aspect of it.

Amos 21:34

Your kids too are, whenever you started working remote, well, did you only have one then? Or did you have two, but they were pretty young. So they've, they've kind of like lived most of their life now with dad being a remote employee.

Chris 21:51

Yeah, yeah. In some form or fashion. Yeah. So that's been interesting too. I mean, but like, but they understand now too that like we can't play video games in the middle of the day, even though they want to do that, we cannot play, we cannot build another Mario maker level right now because Dad's got work to do, but we will later, you know, but what we can do right now is we can eat lunch together, you know, or like I can take Alice to school in the mornings. Like there's all these other ways to be involved throughout the day. But I think you do, there is something about, you know, like you have a job to do, you have to do, you can't, you have to do your work.

Amos 22:27

And do you find- oops, sorry.

Chris 22:29

No, I mean, I that's it really, but you have to, you have to have time to be able to do that without it taking over your life. And so finding the, like when you, when you work, you're working and when you're not, you're not, um, and, and trying, trying to split those two things or bounce between them, for me, super doesn't work.

Amos 22:50

Do you, do you keep a really strict schedule on when you start and when you end work?

Chris 22:54

Not super strict, but I have a schedule, like I'm not militant about it, but I have a very, pretty defined schedule, and don't really deviate.

Amos 23:03

Yeah. I would say that my beginning, my beginning and end of day are I try to keep them within a half hour, like I say 5:30, but that might be five to six. And my mornings too, sometimes just because like, maybe the kids need some extra help. So I try to leave that a little open, but my lunch, I'm terrible about keeping lunch schedule.

Chris 23:26

Yeah. I'm bad about it. And I've noticed that whenever I, uh, I'm always happier when I have a really well-defined lunch schedule. Like when I take a lunch break basically at noon and then come back at one or whatever, you know, around there, sometimes earlier, sometimes later. But I typically, um, typically I'm typically in the office, like in my home office by nine plus or minus 15, 20 minutes.

Amos 23:53

Okay. That's not too bad,

Chris 23:55

Like in ready to work, um, and I, I typically stop it almost right at five, maybe like 4:30.

Amos 24:02

So your, your kids, my kids partially homeschool at least. But your kids go out to school?

Chris 24:10

Uh, yeah. Historically.

Amos 24:11

Okay. So, so I was going to ask like, your kids are just starting, their home now, and you've also worked over summers and stuff, but not everybody who's doing remote has had to deal with that. Like their kids go off to school and they go off to work at the same time. So how, how are you dealing? Like what are the, some of the things that work for you in having the kids at home while you're working?

Chris 24:41

Man. Yeah, it's tricky. It's really tricky.

Amos 24:43

For me, the hardest thing was going, like when I would hear them like arguing in the other room or something, not like just knowing that my wife's got this, she, you know, I don't have to be there every time somebody goes, "Oww!"

Chris 24:57

Right.

Amos 24:59

Cause that was a big change for me.

Chris 25:00

That's definitely, that's definitely really hard if it's really bad sounding, I'll go out there. But honestly, for the most part, they play outside or they're doing school at the table. They don't fight too much, uh, part of that's because we've been pretty honest about, with our oldest, whose typically the instigator of a lot of that. And just sort of told her like, listen, this is a serious thing. Here's what's happened. Here's why you're home. Here's why we're home. You know, here's why we're doing social distancing. Here's why we're doing all these things. And we're not trying to scare you, but this is the reality of our lives right now. If we do these things that will help a lot of people. And by being honest with her, she's not, you know, we're probably either giving her a lifelong series of, of, of therapy problems, uh, which will be treated in some way. But at the same time, it's been really helpful just to tell her kind of, you know, we're not like doom, we're not like spelling doom for everybody, but we're pretty honest. And I think that's really helpful for her to understand when we say like, we need you to do these things. She understands sort of the import of those.

Um, and that's helped actually, uh, keep things- But at the same time, you just sort of, you just learn to tolerate a low grade level of yelling, you know, at any given time.

Amos 26:22

I should get better at that.

Chris 26:22

There's gonna be noise everywhere. There's going to be yelling, you know, their kids, you know, we're like I said, we're fortunate that we have a backyard. They can run outside and go play in and there'll be okay.

Amos 26:37

Yeah. I feel for the people that are stuck in like third floor apartments.

Chris 26:41

Oh yeah. No like that, that's so much worse than what I'm dealing with.

Amos 26:45

I'm sorry for you. And if you want to, you can go out to our website and tell us about your coping coping mechanisms because Keathley and I, neither one of us have had to deal with that.

Chris 26:57

We're not doing anything as difficult as what you're dealing with right now.

Amos 27:01

Yeah. So if you have things that you would think you should share, tweet at us, something we would love to, to be able to share that with everybody. Like, that's my big thing. I'm just, I'm trying to figure out how, how can I help people keep that social distancing? And I did talk to my kids about like, look, you know, we're, we're doing this to protect society as a whole, like, like, I don't think that my children need to be scared that they're going to die. Like, the kids are okay. But it's, you know, we're just, we got to keep the hospitals open and stuff like that.

Chris 27:35

Exactly, yeah.

Amos 27:36

Anybody who's out there, who's listening, who's like throwing coronavirus parties, please stay at home,

Chris 27:41

Yeah. Stop it, ya jerks.

Amos 27:42

And if you want to get on a hangout on, on Google Meet with me so that you can have somebody to talk to, that's cool. Just let me know .

Chris 27:54

Yeah, we should do that at some point.That'd be fun.

Amos 27:56

Yeah.

Chris 27:56

We should do an Outlaw party.

Amos 27:58

Maybe we should do an Outlaw conference online.

Chris 28:01

Let's just do like, okay-

Amos 28:03

We'll start with a party. Let's start with a party.

Chris 28:05

Let's, yeah. Maybe dream big on your own time. Okay.

Amos 28:10

I like to dream big when other people around and hope that they take the reins and I don't have to do anything. That's that's the goal. Then I just do the fun stuff.

Chris 28:20

Yeah. Okay. Well don't drag me into your giant, your giant dreams of like, let's do a whole conference, like, allZ I just wanted like a single Zoom call with a bunch of friends. Okay.

Amos 28:31

Let's -okay. Let's do that.

Chris 28:32

A single lunchtime Zoom call with friends. And you're over here like, you know what we should do, we should make t-shirts, throw a conference, we'll get speakers. Uh, we'll do some sort of send out, you know, gift cards. People can cater their own lunch.

Amos 28:49

Now you're pushing mine to the next level. And you have great ideas. You should totally run with this Keathley. It would be amazing.

Chris 28:57

You're over here, like, like all I wanted to, I just wanted a call and next thing I know I've got a week long extravaganza. I got a gala I got to go to now online.

Amos 29:08

That's right. Bow ties only.

Chris 29:10

Not cool, bro. Not cool.

Amos 29:14

Oh, I think I, yeah. I think a lunchtime chat would be fun. Maybe, maybe record it. Maybe not. We'll see. We'll see.

Chris 29:28

So that was fun. So topical. Lots of, lots of fun news.

Amos 29:33

I think it's good though, that we talk about this stuff.

Chris 29:38

Here's the thing . I'm happy to revisit this, uh, in weeks to come as long as it is, uh, we focus on constructive ways to do remote work better. That's what I'll say.

Amos 29:53

Yeah. Fantastic. But I think, I think we had to just, I don't know. I felt like I needed to talk to somebody other than my family about everything that's going on. So it's nice to see that, you know, you guys are doing good. Um, almost everybody I talk to, everybody's doing okay right now.

Chris 30:12

Yeah. Yeah. Same, same.

Amos 30:14

Good, good deal. So what about Elixir world? What are you doing?

Chris 30:18

Uh, what am I doing? I'm just writing a lot of code. No, just writing, writing tons of code.

Amos 30:24

I have bronchitis leftover from that, uh, from the last conference I went to, so maybe remote conferences are sounding good right now.

Chris 30:33

Um, I broke, I broke a thing in, I submitted a feature for, um, the Kafka library that we use called Elsa, which is really good. Check it out. It's maintained by friends of the show. I broke it because I, in the feature I submitted, the feature just doesn't work, which is typical of me when I don't write tests for things as much as I'm aligned TDD. I always, I, I always get details wrong. I'm bad at that. So-

Amos 31:02

Should have written tests,

Chris 31:03

I, you know what, I did test it. And then I changed something at the end about the API and uh, inadvertently broke it. So yes, so in this case you're right. I should have written tests. I should have written tests to verify that it works. A hundred percent accurate statement, a hundred percent accurate statement. So I'm fixing that, but mostly I'm just writing a lot of code. Like I'm writing just so much code in the belief that I can somehow solve problems that way. Most, a lot, a lot, lots of human problems that you can't solve with code, I'm still attempting to solve them with code.

Amos 31:36

Nice.

Chris 31:39

So we'll see. We'll see how that pans out.

Amos 31:39

So me, me too. Um, I tried to get my wife to let me buy a farm bot, you know, cause I was like, Hey, we're going to need a garden this summer. Just, this is just my excuse to buy new toys. Like.

Chris 31:53

You can just have a garden. You don't need a robot. I'm here to tell you.

Amos 31:57

Shhhh. Don't tell people that. If you say that too loud, she might actually listen to the podcast.

Chris 32:01

Here's the thing. People had people had gardens-

Amos 32:03

She won't let me buy one. Anyway-

Chris 32:04

People had gardens for a really long time.

Amos 32:07

I know. I know. Um, but yeah, so like today's is my, I call it, we call it Noggin Day-n.

Chris 32:17

Long before Ben Ben Franklin flew a kite in the rain, people had gardens Amos,.

Amos 32:22

Wait a second. I thought we didn't start gardening until after-

Chris 32:26

Oh, in a post Ben Franklin world?

Amos 32:27

Well, I thought it was a post-Civil War world.

Chris 32:30

He invented the bifocals and uh, the notion of growing vegetables.

Amos 32:35

Yeah. Before that everybody was just hunting and gathering.

Chris 32:37

Discovered electricity, invented the bifocal, I believe had syphilis, and uh, you know, invented gardens .

Amos 32:48

And, and those things that you put on your feet.

Chris 32:49

Early to bed, early to rise, also plant carrots.

Amos 32:49

And the things that you put on your feet when you swim. What are those called?

Chris 32:56

Flippers?

Amos 32:56

Flippers. Yeah. They said, I just read somewhere that he used to strap planks of thin wood to make his feet longer and he would go swim nude. I mean, outside of-

Chris 33:09

As you do.

Amos 33:09

Yeah. I mean it sounds like a Benjamin Franklin thing, if you've read a lot about him. So yeah. So today's our, well, we call it Noggin Day at work, but it's just our day to do fun things. So I'm working with Connor this morning, looking at some Scenic stuff. Uh, I've been working on a lot of builds lately and you know what I can, I really appreciate actually the mix release stuff thatcame out way more, because, you know, I did everything with e- deliver before and I like e-deliver, it's got some, some good features, but I feel like mixed release, just simplified my life. And then I'm actually using circle CI for building. And as Connor likes to say that they give you free CPU, free build time. Just use it, like

Chris 34:00

So much free CPU. Yeah. Well, this is, this is relevant to my interests because I am currently, uh, working on a thing where I, uh, unrelated to work just for fun. Something maybe we talk about in the future, where I, uh, I, I have the need to, to provision and build some, uh, some artisinal hand-rolled Bespoke, uh, servers and deploy code to them.

Amos 34:23

Nice.

Chris 34:24

So tell, tell me, tell me how it do Amos.

Amos 34:27

So, so the circle? Circle CI?

Chris 34:28

Nah, just talk, talk me through a deployment process that doesn't involve the words Kubernetes, Docker, all these things. I want to, I want to hear it.

Amos 34:41

Oh crap. Okay.

Chris 34:41

You're using the Docker.

Amos 34:42

Well, because Circle CI, we build, we use Docker to build the box that actually-

Chris 34:47

Okay. Fair enough. Fair enough. This is what I'm interested in. Okay. Tell me about your process.

Amos 34:54

We have an article about it that Connor wrote too on the website. Um,

Chris 34:57

Tell me how it do.

Amos 34:58

So basically whenever we push a tag, is what we have it set up for, so whenever there's a tag on, on a production branch, Circle CI will pick that up and we have some Circle CI config. And we also have, we also have a Docker container already out there for you, for working with Elixir that we linked to in the article. Maybe we'll throw this in the show notes, but if you go to GitHub's, Binary Noggin's account on GitHub, you'll find that there. So we have the circle CI set up to use the Docker container that we have defined that has Git and Elixir and Erlang and all the stuff that we need. And it picks up our project only if you tag it, and it runs mixed release. And we always go to the steps on mixed release, and you have to have a symbol in your list of steps if you define them. But then after that, there's another one, tar, which is a beautiful thing. And it tars up the project. And then, well, after it builds it, so it builds a release and then tars it up. And then we have the keys, Circle CI has keys that connect to our production servers and it sends the TAR over there, tells it to un-tar, calls stop on the process and start on the process on the new one.

Chris 36:17

Got it.

Amos 36:18

And it takes a pretty short amount of time actually.

Chris 36:20

Right. Yeah. Especially if you have a lot of this stuff already, like if you've got a cached, like building the releases really quickly or really quick.

Amos 36:28

Yeah, yeah, yeah. And we do, we set up Circle CI caches for dependencies and stuff like that. So it doesn't pull those down again and, and I'm sure that there's ways that we could streamline and make our build better. Um, but it, it works pretty well-

Chris 36:46

For what you're actually doing.

Amos 36:47

Yeah. And then you have the release configurations are nice because they do give you some sort of like start time configuration for-

Chris 36:57

Eh, we just, we don't use any of that.

Amos 36:59

Um, I think, uh, I might be using it for one thing. And one thing only, and its a gem, or not a gem, a package.

Chris 37:09

You, you listen, you gotta do what you gotta do. And I do understand that you've got dependencies that rely on mics on config files. You know, you gotta do what you gotta do. Cause people aren't aware, for releases, you have a, there's like a specific like release. exs file that you can add that will, uh, inject stuff into your application config. And it does it after the, after the app boots. That way you can populate it with like runtime variables. We largely skip that entire thing. And we use Vapor for all of it. And we just, the very first thing that happens when your appli- in your application start, you know, Def start, underscore underscore, cause you never need that crap. And uh, and then you, uh, the very next line is typically always like vapor dot load bang, and then load the configuration that you've specified.

Amos 38:04

Yeah, I haven't, I haven't used Vapor, but I can tell you, I used Tomal before this and that was horrific. And I think environment variables seem to work pretty well, if you only have a couple.

Chris 38:19

The benefit to Vapor is you can use whatever you want. So you can use JSON files or Yammel or Tomal, or.

Amos 38:28

Aaah!

Chris 38:28

Or actually Tomal is way better. Thomal is way better than Yammel or JSON, like way better. Not even close.

Amos 38:39

Yes. I would agree. The thing that I like about the exs is that it's Elixir. And so I don't have to change my brain whenever I'm looking at different files.

Chris 38:49

Sure. The problem with releases and the reason we use Vapor, uh, amongst other things it's really easy to add, um, it's really easy to do conversions because you can map, you can, for every binding you can basically say like, here's the environment variable I want , put it into this actual variable in the map, in the map you're going to give me at the end of it. And then that's called a binding. And when you have a binding like that, you can say like this has to be required. They're all required by default. So if they're missing it throws or you get an error back,so that the app won't boot,

Amos 39:26

That is nice.

Chris 39:26

Which is, that is exactly what you want in literally, almost every case.

Amos 39:31

Right.

Chris 39:31

You, you, you do not want to boot the app with a missing piece of configuration. Along those same lines, you almost never want defaults. Like, I don't think you ever want to use a default value if you can help it, because here's the thing, default values and configuration, they only help the developer while you're in development mode. Get over yourself. Like just set up your environment correctly, the way you would with production and always get the variable, because what will happen is you'll deploy, you'll not have the variable there, it'll fall back on the default. Everything will look correct. And it won't be, and then you'll get weird subtle bugs.

Amos 40:15

So I, I, in the past, because of that, I've, I've set default, default values because they had to be there, but I set them to like, database not configured, so that when it actually blows up, I see a string that says database not configured. And I'm like, oh, crud.

Chris 40:33

So Vapor forces everything to be there. It gives you translations like really easy to use, like you can just map, call like string.to integer on stuff, to convert it into integers and all that stuff.

Amos 40:44

When you, when you run Vapor, does it give you some kind of a config map back or something that you pass out down to the other stuff?

Chris 40:50

Yeah. It just gives you a map.

Amos 40:51

Okay.

Chris 40:51

Gives you a map with keyword lists.

Amos 40:51

Is there anything built into vapor for, um, configurations changing while the system is running? Like notifications?

Chris 40:59

No. That that does not exist. Um,

Amos 41:04

It's a hard problem.

Chris 41:05

And for a long time, Vapor's an interesting project for me because Vapor's the project I've struggled with the most to come to the right API. And I think it's finally pretty close to the right API. Although I think that there's- what's the right way to say it? I think that the building blocks are all there and now it needs an actual, like, easy to use API over the top of it. But all the building blocks are there to build that now. Cause it's just all structs and Elixir data structures. For a long time, I attempted to build an API that would do things like watch configuration and then update somehow and then emit that update to everybody. But the downside of doing that is one, it's a highly stateful thing, that's really easy to get wrong. Uh, two, you have to care about all these different errors semantics, and because you're caring about them, you're forcing everyone who uses your library to buy into those semantics, and that is often not really what you want because everybody's got, every service is going to have different ways to talk about this stuff. And so at the end I just ripped all that out. You know, it's like good design, right?

It's taking things apart and pulling them all apart and just doing one thing really well. And so Vapor's really only focused at like loading the right variables. And if you want to load stuff from external sources or if you want to watch it and load it on demand, like you'd pretty much have to do that yourself, but then you're in control of it. So like the hard part of getting this layered together map of stuff where, you know, it's all there, that we do for you. Putting it in persistent term or putting it ETS or, you know, notifying processes to restart with new configuration or whatever. That's not really a thing that I think we can universally solve because you just have to buy into it too much. Like so many other things I have to buy into that process, if you do it that way, that it makes it less useful. So you'd have to do that yourself. But that I, I find a lot of benefit in doing it that way, because it makes your development process much closer to how your app actually runs in production, because like release. esx doesn't run when you're doing local development. Some other config file does. This way, its like, basically we don't have any, we don't even have a release.

exs file. And typically we really only have like the, the config files we do have, you know, there's maybe like, I don't know, 15 lines of code between them when you cat them all together.

Amos 43:37

Oh, that's my bad then.

Chris 43:38

Yeah. It's like, you can look, and it's often, it is stuff like we're injecting certain types of modules into them, right. It's the stuff that you actually want to do in config. Like not compile time necessarily, kind of compiled at a time stuff, but often it's, you know, things like inject this mock module into the thing or, you know, don't actually try to call this real service while you're in development mode and that sort of stuff.

Amos 44:03

Okay. Yeah. Yeah. Those are, those are probably a lot of the things that I see using configuration for anyway. So it is nice to have that as some runtime thing. And then I, the thing that I like most though, that you said the whole time there is that it requires everything to be set by default like that makes me happy. Like I don't find out, I don't find out 20 minutes into my system running that I forgot to configure something.

Chris 44:30

So you can add defaults to the bindings and you can mark bindings as not required. That is possible to do. By default, everything is required. And it took me a long time to add that. In fact, like, basically Fred pushed me over the edge on adding it. Cause he was like, people are going to do it no matter what, they're just going to work around you. So you might as well like just support it. Like they just basically won't, they'll just ignore you, so you oughta support it because.

Amos 44:59

Sounds like IBM in WWII.

Chris 45:00

With a giant, you know, with a giant, uh, with a giant asterisk, like you don't really want to use these that often. So that's my asterisk, is don't use it that often, but they are there if you want them. So. You just don't want them.

Amos 45:12

Do you have a giant asterisk in your, um-

Chris 45:13

I do.

Amos 45:14

In the documentation?

Chris 45:15

Yeah. I believe there is.

Amos 45:18

Here's how you do default arguments, but don't.

Chris 45:20

Yeah. I want to go back to this, to the deployment stuff. So you build the, the release in Circle, you push directly to the server.

Amos 45:30

Uh, yeah, I pushed to, well, we, we are the one that we have, um, it uses a Digital Ocean.

Chris 45:36

Yeah, right. Yeah. I also ride the Digital Ocean wave.

Amos 45:39

Yeah. I mean, for, for my projects internally, five bucks a month. Fine. Great. Um, so we, we push out to the Digital Ocean server. We un-tar right in the home directory cause like every project we just give it a user. So we un-tar right there in that home directory. Um, and the, the un-tarring it has a, a fantastic thing is that there's a releases folder that has every single one of your releases that you've un-tarred before in there-

Chris 46:09

Oh, fancy!

Amos 46:10

-under their own folder.

Chris 46:11

And that's just so that it just, they just, you did that or you did that or it just does that

Amos 46:15

No, no, whenever you un-tar, so it normally does, like, let's say you have a release zere point one point zero. It will put a releases directory. And then inside of that, it'll have a release zero point one point zero directory. And all of that stuff goes in there and then there's a start Url dot d- that file, that data, that dat, something that just kind of tells it what version to start out of all of those. So that means like if you have to roll back to a version, for some reason you just go change that start url.data file. And that gets- and it'll, and then restart your server. I haven't had to do that. So we don't have anything automatic set up to do that. The other thing is you could just redeploy an old version if you wanted to. But that takes a little longer than actually just changing that one file and restarting.

Chris 47:06

And then are you, are you running all this under like, like supervisor or is it, or like system D or whatever?

Amos 47:12

I should be.

Chris 47:15

But you're not, you're just running it or like Upstart or something?

Amos 47:16

Yeah, I should have made some kind of Ubuntu, it's, we're running on an Ubuntu box.

Chris 47:20

Oh, okay.

Amos 47:22

And should probably have done some moving to service level thing, I believe right now that-

Chris 47:28

Well then you just stop the exits- the running one ,and start the next one.

New Speaker 47:31

Right. Yep.

Chris 47:33

And then it restarts. So there's like some downtime there, but whatever.

Amos 47:36

Yeah. Uh, I am trying to think, I think there, there are ways to not do that. Like you can tell it to reload, I think, but.

Chris 47:46

You can SimLink in there and you can also, there's a bunch of ways you can, you can kind of do the zero downtime thing, but what you're describing, it's like it's whatever.

Amos 47:55

Yeah. Um, I think that if I wanted to take it to the next level, so what I do with some clients is , do like a red green deploy. We actually deploy to a completely new container, new server. Um, maybe you have two of them out there and then a load balancer in front of them. So you spin one up and then run some like basic little "Is the system actually up and running?" tests. And if they are, then you switch the load balancer over. Um, that I think is a good way if you don't want to have to do upgrade deployments. So I'm not doing hot reload. Yeah. Um, so that's a good way to get away with not doing hot reloads. Um, and it gives you, what I like about that version versus hot reloads. I think hot reloads are awesome when you need them and you should learn just enough about them that if you ever need to do them there, that you have some knowledge there. And I think you should try it. And I've, I've done that like, like a year ago playing around, but not with the new mixed release. I don't even know if you can.

Chris 49:02

I don't think you can. I don't believe that's built in.

Amos 49:04

Okay. Okay. But the thing that I like with the red, green, red, blue, whatever you want to call it, deployment.

Chris 49:10

Blue-green.

Amos 49:10

Yeah. Is that-

Chris 49:12

Black and yellow.

Amos 49:14

One- black and yellow, black and yellow._ Um, anyway, uh, um, one of the, the server that you're bringing up, if you want to, you don't have to switch for awhile. You could actually like go in there and play around with it and do more extensive testing and then switch if you need to, um, you could treat it as like a staging, if you wanted to, as long as your staging, doesn't create a bunch of crap data that you need to delete. It, it could be used as a staging that can be tested by QA , or yourself, and you could performance test on it. Like if you're working on performance, you could say, okay, now we have a production server up. Let's see what the performance on that box is versus what it was on my laptop. And then whenever you say, okay, we did make an improvement now switch to a load balancer. So the, um, so we, we do that. I do that with a client and it's, they're using AWS and AWS has a lot of stuff built in for that, where it'll spin up servers and wait five minutes to make sure that they're actually handling requests and not getting a bunch of errors.

Chris 50:19

Right. And tear down the old one, and all that stuff. Yeah.

Amos 50:20

So like, you can get super complicated with it.

Chris 50:22

Right. Yeah. Yeah. But I mean, for, for my purposes, I just need a couple of Bespoke, uh, DigitalOcean boxes. And, uh, you know, just for when we'll slam, you know, some, some releases out there.

Amos 50:35

Right. For $5 a month.

Chris 50:38

Are you running something like an engine X as like a reverse proxy for your, for this stuff? I mean, I'm assuming these are web projects also. Or something that has an API.

Amos 50:47

They, they are- all of them out there right now are web projects. Um, because it's a playground for me. A lot of times, some of them are not like my invoicing software is written in house, um, and deployed out there, but some of them are play projects. Um, and right now, yes, they are all web projects. And I do use engine X to proxy into them. Mainly like dealing with HTPs and, and that way-

Chris 51:18

And like Encrypt and whatever else.

Amos 51:20

Yeah.

Chris 51:20

Yeah, yeah. That's, that seems like a reasonable stratedgy.

Amos 51:24

And there's some other neat stuff that you can, and you can do some other neat things with engine X, if you need to. Like, like, if I actually wanted to take the app down for awhile, I can have it automatically switch over on engine X to a, uh, uh,

Chris 51:40

Static site or whatever.

Amos 51:43

Yeah, um, and, and, uh, I have not done this, but I've, I've talked to somebody else who used engine X almost like a load balancer. Um, so it's nice to just already have that in front so that when you want to add those things, you can, um, I also found dealing with SSL, for my brain, I've done it for so many years in engine X that I can just go do it in 10 minutes. Versus trying to put keys into the Elixir app. And like, I don't even know how. And I don't care to know how. Cause I don't need to.

Chris 52:16

Yeah. Yeah. I know what you mean.

Amos 52:19

And then, uh, I was told recently, I had no idea, but there's a way to, if you're using Let's Encrypt and engine X to have engine X automatically get those key updates. I had no idea, like mind blown. I was talking about writing that and somebody is like, oh, it's already there.

Chris 52:38

Yeah. So it was like, you just do it.

Amos 52:41

Yeah. So I, maybe I should do that today too. Part of my Noggin Day.

Chris 52:45

It's very fun to go back to, er, to kind of go back to doing a lot of this stuff I have. It's been so long since I've actually had to care about this, uh, you know, for work, we use Docker and whatever. Just jam giant hosts filled with Docker containers. And that's fine for the thing I'm working on. It's like, I need like two things. It gets silly to, it'd be silly to try to like replicate some gianormous complex beast of a deployment strategy for two servers, right. It's already over engineered. Here's, like, here's the thing, right. Like me being me, you know, I'm using Terraform to like scaffold out stuff just cause.

Amos 53:24

I hate Terraform.

Chris 53:26

It's you know what it's, it's really the worst- like

Amos 53:30

I don't know that I can find anything thats a whole lot better than Terraform. But I hate Terraform.

Chris 53:35

That's the problem. Terraform is the Kubernetes, uh, which is the JIRA of tools, which is to say like, just because you're the most popular doesn't mean you're good. And, you know, Javascript

Amos 53:43

Next time you have to do a for_each loop Terraform, you just get, gets makes me angry.

Chris 53:49

Oh no. I mean, all of it's bad, but I do like the ability to sort of just like tear down and rebuild this stuff basically from the command line, that's pretty rad. Like the thing that it allows you to do is rad. I'm also using another one of my hated tools, which is Ansible, for like configuring a lot of stuff. Uh, it's fine. It's just configuration management, but it's like highly. It is, um, highly imperative, highly imperative. Just lets you run, run commands on over SSH on boxes, on multiple boxes.

Amos 54:23

I like thatAnsible automatically allows you to communicate everywhere instantly.

Chris 54:29

Is this a reference?

Amos 54:32

I was trying, I was giving you a chance to explain the reference.

Chris 54:36

Naw, I don't want, I ain't about that life. But anyway, so I'm doing this because that's already, that's already like way more complicated, like you shouldn't even be doing that for spinning up what amounts to like two servers and a database. But I'm doing it because-

Amos 54:48

Why are you spinning, are you, why are you spinning up two servers?

Chris 54:51

I got stuff I gotta do a dog.

Amos 54:53

Are they both running the same thing?

Chris 54:57

They're running stuff.

Amos 54:57

I was like, I thought maybe you were like throwing it a load balancer in front of your little play project. and I'm like "What? "

Chris 55:03

I mean well I got some customers coming up, but whatever.

Amos 55:05

Nice, nice.

Chris 55:06

Anyways, the point being-

Amos 55:08

I think you needed a load balancer as soon as you have enough customers that they actually pay for it.

Chris 55:12

-It's been a minute since I've had had to do this stuff. Um, and so it's a chance to go and relearn some stuff. It's good. It's good for you. Yeah. Good for you.

Amos 55:23

It's a, you also get to relearn all the frustrating parts of it though.

Chris 55:26

Yeah. I mean it is, it is one of those things where you're like, oh right, yeah, this is, I remember now why we don't do this anymore that often. Like I remember why. And it also, you know, the really funny thing is it makes me realize just how ridiculously under, uh, under priced like Heroku is when you really stop and think about all the things they're doing for you.

Amos 55:47

Oh yeah.

Chris 55:48

And you're like, this costs me nothing? Like actually nothing. How is that a sustainable business model? Like you know, I guess it's getin' bought by Salesforce, but anyway,

Amos 56:02

I don't know. So why Digital Ocean over Heroku if Heroku's that easy?

Chris 56:07

Oh, I mean, just cause. Why? I mean I could, I could turn around and ask you that question. I don't know. You want to come over here and mow my lawn?

Amos 56:19

If I can tell you what to do, I will come mow your lawn.

Chris 56:22

Oh happily. As soon as the quarantine is lifted.

Amos 56:23

As soon as quarantine is over, I'll be there. It might be next year, but.

Chris 56:30

I mean, most likely. But so no, I mean, we're doing it. I, um, Heroku should, if you're building a real thing and you want it done today, uh, you should use Heroku. It is the like, and I don't work for Heroku. I got no kickbacks from this. They're not a sponsor of the show.

Amos 56:49

They should be.

Chris 56:49

Because there are no sponsers on this show.

Amos 56:50

They should, , they should, Heroku, can you go put some money in our Patreon?

Chris 56:54

Right. You know, I mean, just really just a, you know, cross my palm, Heroku, ok? Wet my beak. I get no kickbacks from this, but like using stuff like that, right? Like you should, 100% be doing that. That's like a no brainer. There's no- And, and forget all the people who are like, well, you can't use distributed Erlang if you use Heroku, or you can't use hot code upgrades, if you use Heroku. And I'm like, yeah, but like, you don't want to do that anyway.

Amos 57:23

But its free.

Chris 57:23

Like, you're, you're, you're literally complaining about a thing that you don't need. You have no customers anyway. Just want to put it on Heroku, just run it.

Amos 57:32

So why?- Okay. So Digital Ocean, you put the question back to me then, right? Why Digital Ocean over Heroku?Okay. Um,

Chris 57:39

Come mow my lawn.

Amos 57:39

Yeah. I don't get on, no, I, so for me it was like, okay, I can pay nothing. And sometimes when I go to hit the app, because it doesn't get a lot of traffic or it takes a while to come up, right. Like it's super slow because it has to actually start it and everything, and Digital Ocean is just always there. I also like being able to SSH in and do some other crazy things that I can't do on Heroku- Like, it's a lot easier to cut a release in SSH in, and then run app name remote and get a console. Um, and so for me, like $5 a month, I will pay $5 a month to have slightly quicker response times. Um, because my app doesn't get hit enough to stay in, in Heroku's hundred running cycle and SSH again.

Chris 58:33

Yeah. Yeah. That's fine.

Amos 58:35

But there's a lot of stuff that I also, like, I, you know, I should put this into the cost. Like you said, there's a ton that Heroku does, even at the free level, you don't have, that you don't have to deal with, so like.

Chris 58:49

Just logging. Just collecting logs, like, it's a huge deal. Right. You know, that's partially to kind of why I'm, I'm personally doing it is because it's been a long enough time since I, you know, provisioned servers and I've never been good at ops stuff that I feel like that's an important thing for me to learn.

Amos 59:11

If you're really good at ops, you probably wouldn't be really good at distributed systems. Like you have to choose.

Chris 59:17

Yeah. You know, it's all, it's all trade offs, right. So I feel like this is an important, this is an important learning process for me, which is kind of a big reason why I'm forcing myself to go through it instead of doing the, the right thing, which is just to get it deployed.

Amos 59:32

So, so have you used GigElixir?

Chris 59:35

Uh, nah, man. I mean, I think it's actually probably a pretty cool service. I know people use it. Um, it's cool. I I'm, I'm, here's what I'll say. I'm really happy. It exists. I'm really, really down with the idea that it exists. Like that's really cool and it's really kind of amazing that anybody built it in the first place and got it running and working, like, that's also awesome. Like, like hats off to them for doing that. That's a ton of work. Like it's just a huge amount of work to get all, to get that stuff up and running.

Amos 1:00:09

And I think it's just like this side passion project from my understanding too.

Chris 1:00:14

Yeah, for sure. And it was basically because of like, just try to solve the, you know, the, the stuff with Heroku to have a Heroku like thing, but that let you do fancy Elixir stuff.

Amos 1:00:24

Yeah. And I was going to ask is, is I thought that I had heard that it's like Heroku where you just kind of push to a, git repository and move on.

Chris 1:00:32

Yeah, I think it is. I don't know. I haven't really looked that deep into it for my own- I have nothing against our service and I think it's probably really cool. And if, uh, you know, I was inclined, I would probably try it out. I think for me, I either want to not think about it at all, or I want to think about it. I want to control it. I don't know. That's a weird, it's a weird, uh, yeah, it's a weird dichotomy in my own head. I either want to not ever care about this in the slightest, or I want to care about it a lot.

Amos 1:01:08

No, I feel that way too. There's a little bit of that thing on Heroku of like, I don't really know what they're doing.

Chris 1:01:16

No. Yeah. And I don't, I mean, there's part of it, like you don't actually really get to .

Amos 1:01:20

And there's probably a little bit of, I mean, there's a little bit of that at, at Digital Ocean too. Like I'm not picking up a box and putting it in a rack and like, so-so there, there's still a trade-off, but I feel like it's a little less, but I agree. Like I really also like the set it and forget it, mentality of Heroku that Heroku allows you to have like, and I'll just turn up the dinos. If this thing actually takes off,.

Chris 1:01:46

Grab the dinos slider, drag it to the right until it starts working well, you know. And you're gonna see a pretty huge performance penalty for, for doing that, and uou know, maybe that's fine. I think, I think your bill would need to be very high, way higher than people think, before you start to actually break even on running your own infrastructure. Unless you have the skills to do that, which I personally don't, which is why I'm doing it, because I think that's like a useful skill to have, or, you know, it's funny, like we were, this came up the other day. I've been, one of the main things I've been working on for the last couple of weeks at work is rewriting, essentially rewriting, porting, um, improving, a bunch of Ruby stuff and moving it to Elixir. And it's just staggering what you can, what you can do still to me, like I've eliminated, I'm pre I did some rough math. And just in terms of what it takes in, in terms of hardware it takes to run the completely equivalent thing in Elixir, we're going to save about $6,000 a month.

Amos 1:03:05

That that's, that's amazing,.

Chris 1:03:07

Right? Like-

Amos 1:03:09

You heard it, you heard it here. This is free advertising for the people who want to sell to their company that they should switch to Elixir. Here you go. This is it.

Chris 1:03:17

We're replacing, we replaced, I don't know how many servers, it's like a single task, like a single Elixir instance replaced like some untold number of servers. And is it is I, it has way better performance still and we're not done. Like I've got a whole bunch of these things to like port over. And if, if current trends keep continue based on our, what our performance we're seeing right now, super back, super, super back of the napkin, math on this, right. But I think you're looking at a lower bound of like $15,000 a month.

Amos 1:03:58

Wow.

Chris 1:03:59

In terms of savings, like basically just from changing run times. That's $180,000 a year. Like, and that's not even including, that's just hardware that doesn't even include operational, like user like me operating at costs, right. The time, amount of time it takes to operate it, the amount of time it takes to deploy it. Like it doesn't account for any of the other, uh, accumulated costs that, which actually those costs completely dwarfed the hardware costs. The human costs are way higher. And now you're talking about just reducing all of that for everybody.

Amos 1:04:40

Right. And like, well, even if you just looked at the, the hardware costs, you could even say, I'm going to hire an engineer that this is their whole job. And they should save me more money than what it costs to pay them to do that.

Chris 1:04:53

And that's like, and it was like three weeks worth of work.

Amos 1:04:55

Wow.

Chris 1:04:57

Like, right. That's like the best dollar value ever. And so it is, there is sort of this, it is kind of amazing. I think if you can do things, like, if you're good at the ops stuff, right. Which I like as we've established, I am not. But if you were good at that, there's a certain amount of money you can save, assuming you have all those skills, assuming you have time for all that stuff, right. You know, assuming you have a team large enough to support that, right. You can save a ton just by running like a really big box or like two really big boxes and just doing your own ops on it, right. Cause like a really big box, a really big Digital Ocean box, is like what', 20 bucks a month, and you're going to run two of them. So 40 bucks a month, you know, and then you just do it all yourself. Like you can, you can save some coin doing that. Non-trivial coin. That's not gonna be worth it for everybody. A lot, most teams like don't have the time or in my, in my experience, a lot of teams don't have that sort of operational expertise to be able to do that and, and really do it well. And so it's worth your time just to like use something off the shelf, but it's pretty wild.

Amos 1:06:12

And then selling that to the company. The other thing that I see with a lot of teams is they have, they have too much feature crunch to ever get the time to do that.

Chris 1:06:21

Yeah, exactly.

Amos 1:06:22

One of the, one of the big spaces for customers that I work with that are trying to convert from like Ruby to Elixir is I see a lot of like jumping through hoops where they're trying to do, I'm going to call it distributed map reduce. And I don't mean like distributed across multiple servers necessarily. But being able to distribute across multiple processes, your map and your reduce. So like something that fans out and comes back together. It's really hard to do with, with background processing libraries. But, but you can use something like flow and have it done in 10 minutes in Elixir and still be way more performance.

Chris 1:07:02

Right. Right, right, right, right. Yeah. It's pretty wild when you really start. And I mean, obviously we're biased, I'm biased. The company is biased because we have a lot of expertise on running Elixir systems and doing tracing and production and diagnosing production problems and fixing them and moving on and all that kind of stuff. So we're biased in that way and you can get good at debugging anything, you know, people run like giant Perl systems out there in the world and they're good at it. You know, you can get good at diagnosing and debugging running systems and any assorted tooling. That said, the amount of tooling you get for free in this runtime to be able to like, make your systems operable, meaning you can get into them, you can trace through them, see what's going on, fix problems in production. You know, it's just, I don't need, I'm so broken. It's like, I look at other runtimes and I'm like, you can't do anything. You all, you're just so handcuffed on what you're able to do in a live running system. It's pretty wild, like what you just get for free. Yeah. I'm, I'm all, I'm just constantly staggered by it.

When I, when I really step back and like do the comparison, it's pretty wild.

Amos 1:08:24

We need to, we need to get some more conversion stories on here. Cause they excite me. Like I just, I see it every day at work, but I can't always talk about all of it. So we should find some people that get to, that can talk about it.

Chris 1:08:38

Yeah. For sure.

Amos 1:08:40

Well, we've been on here. We've been on.

Chris 1:08:41

This is going long. We've got to go.

Amos 1:08:43

It is going long. Um, I just want to say everybody out there, stay happy, healthy, and safe. Please stay home. Find, find some community online. Um, hang out, reach out to us. Um, maybe we'll see if we can help you out. I mean, I don't think we have enough listeners that we 'llbe completely overwhelmed with people reaching out to us. If we are, I'll make Juliet talk to them. Uh, anyway, um, I just, I just really wanted to say that everybody say happy, healthy, and safe and, and we love you and miss you. And we'll be back at conferences when they, they get going again and see everybody and in face in person to person.

New Speaker 1:09:21

Take care everybody. Alright later Keatley.