Over Engineered

00:06.22
Chris Morrell
All right welcome to another episode of overengineered the podcast where we ask the very important question. What is the absolute best way to do things that don't really matter. Um I am here today. Ah.

00:21.80
Tim MacDonald
I'm sorry I Love the premise of this.

00:24.67
Chris Morrell
It is I like the fact that I get to do this is so much fun. Um I'm here with my friend Tim Mcdonald um my internet friend hopefully ah real real world friend after leracon that will be fun. Um.

00:27.66
Tim MacDonald
Yeah, definitely.

00:38.11
Tim MacDonald
Yes.

00:41.54
Chris Morrell
And before I introduced him I realized last episode I never introduced myself. My name is Chris Morell and I am I'm a ph b developer I've been writing php code for twenty plus years and just thought there are lots of topics that. Come up ah over and over but never feel important enough to really give a lot of time to and we just thought let's sit down and really ah, arguably overthink them so without further ado ah tim, do you want to introduce yourself and and honestly I don't I don't know like what's ah, what's your give us the short version. What's your what's your story how do you get to where you are right now.

01:22.93
Tim MacDonald
I Mean the short version is I'm just a random php developer. Ah, that's that's it really? Um, how did I get to where I am I started off like as far as development stuff goes started off doing like wordpress stuff. Worked it like a Wordpress factory or it was just like website after website after website um during that time I googled What's the difference between javascript and jquery to try and understand that world because I just didn't understand what that even Meant. Um.

01:44.54
Chris Morrell
Nice.

01:51.70
Tim MacDonald
And then kind of started looking into the app world and building apps and at one point I was heading down the path like I built some ios app and Android apps and I was heading down that kind of mobile path and I was like and the web's kind of cooler. Want to come back to the web. Um, you know, found laravell fell in love with that build lots of apps. Ah, and yeah here I am still just kind of writing page p and loving it.

02:14.65
Chris Morrell
And now you are on the larave the larave team right.

02:20.72
Tim MacDonald
I am yeah yeah, which is which has been amazing. it's ah yeah it's it's really cool. Ah not quite a year yeah so coming up to a year I think soon which has just flown by that's absolutely nuts to think about. But yeah, coming up to a year I think.

02:24.96
Chris Morrell
Um, how long has that been. Okay.

02:37.58
Chris Morrell
Wild nice all right? Well let me part of part of why I am really excited to have Tim on is I was talking briefly about the concept of this show and I said for example, here's here's ah, here's the thing that we might talk about right.

02:38.81
Tim MacDonald
Um.

02:55.23
Chris Morrell
You've got ah your you've got your migrations and in Larell, there's a very straightforward story for for schema migrations. But what do you do when you need to manipulate data in some way that's not part of the schema migration right? Do you just like. But the example that we gave last episode just briefly is like you're doing sort of a rolls and permissions feature and you know you've just migrated a roles table you've just migrated a permissions table and you know at the base level. You're going to need to create a couple of roles and a couple of permissions to get started and it's not. Those are the only roles and permissions that are going to be in the system but you know like immediately I need you know a couple of a guest role and an admin role and a staff role or something like that and I need a couple other things and. Ostensibly maybe the feature that you just pushed won't even work unless there's some minimum amount of data in the in that that sort that database that you just created. So yeah, we explored a little bit of like the different ways you might do it. Do you just throw it in the migration directly. Do you write a customer. Command that you have to then ssh into your app app server and run after the migration or a couple other things and we're going to. We're going to go come back to them in a couple things that folks said on like Twitter and mastodon. But what really intrigued me was Tim said I've got an idea and.

04:24.31
Chris Morrell
It's not anything that I've heard anyone else say and so I've been very curious to hear what this idea is um so before we get into any of that like let's hear it I'm I've been waiting weeks for this.

04:38.15
Tim MacDonald
Just to jump straight into it. Yeah, so as you mentioned Chris like you kind of planted the seed that you wanted to chat about this with with with developers and um.

04:48.47
Tim MacDonald
That day I was riding home on my bike and as all good problem solving like that was when I started to like think about this problem we dig deeper and kind of like start to pick it apart and I was like oh I wonder if you could do Xy and z because we had something um, come up with larille forge recently that. Kind of helped me kind of put all this together on the bike ride home that I think there's like a deeper level to so I think that as laraville developers we've all kind of thought about this problem right? because we've all had to tackle it at some point once an application kind of gets down the line far enough. You know you need to you need to do something like this. So. It's something that I think.

05:18.23
Chris Morrell
Absolutely.

05:24.36
Tim MacDonald
A lot of us have put some kind of thought into and I myself had put lots of thought into it and then yeah it was just that ride home something just clicked and I thought oh this could be cool now full disclosure. This could be an absolutely terrible idea that would never work and has a big hole in it and you're going to point it out the first 2 bits of this podcast. So like that's it. But.

05:40.60
Chris Morrell
I Can't wait.

05:43.49
Tim MacDonald
But ah, my idea is laravell envoy essentially. Okay, yeah so Laravell Envoy is a tool to kind of have deployment scripts for your application I think we should define the problem like.

05:49.73
Chris Morrell
Okay, say more.

06:03.31
Tim MacDonald
I know you've kind of talked about the problem in the previous podcast but because there's a few things right? like you want to solve the different problems in different ways. Um, and so 1 thing you talked about was the migration stuff now when I think about the migration stuff. There's a couple of things one is maybe I've just added a new column. To an existing table and I don't want to so I ah I think you need to solve and and say it's like a column you want to add you want to populate quite often I find myself wanting to add a column populate it and then make it non nullable. So.

06:23.51
Chris Morrell
Yep.

06:38.40
Chris Morrell
Yep yep for sure. Yeah.

06:39.77
Tim MacDonald
You want to make sure that it has all those values in it now. There's no point in writinging a migration that adds the thing then riding a cedar that populates or you can write a cedar class but like running the cedar as a separate process and then running another migration to like make that non-nullable and I think in that case if you're kind of if you're tacking on a a column.

06:51.64
Chris Morrell
Right.

06:58.40
Tim MacDonald
Then What you want to do is is add the column just loop over all the rows add you know, populate it there and then make it non-nullable. Um I guess depending on the the. Size of the database. Maybe that isn't Practical. You do actually need to kind of do that in the background if you can if you can like launch the app and it can run without those values there. Maybe it kind of has to be blocking. Whatever but I think something like that just has to be in the Migration. So I think to answer that particular problem I think that is like do that in the migration that's kind of my opinion for.

07:29.53
Chris Morrell
Yeah I mean like all things right? It depends and and I do think that there are a lot of cases I mean and yeah, this is where it comes up is like yes, but what if that table has 10000000 rows in it.

07:30.98
Tim MacDonald
If as long as it has to be blocking. Yes, Yes, oh yeah, yeah yeah.

07:49.31
Tim MacDonald
Um, and all the pair.

07:50.70
Chris Morrell
Um, and you know realistically going back to the it depends. The answer is well when you have to deal with 10000000 rows maybe you have to have a different solution. But yeah I'm I'm with you so far.

07:58.66
Tim MacDonald
Yes, yeah, absolutely absolutely and and you know a perfect case. Scenario is is you probably like add that column before you add before you deploy the feature then you like populate it in the background and then you make it non-nullable.

08:13.44
Chris Morrell
Um, right.

08:17.00
Tim MacDonald
And then you deploy the feature when it's already kind of populated and and pre-filling itself and kind of do like a staged migration or something like that Sosa. But okay so I'm jumping all over the place I'll I'll get to the the envoy stuff. So my thinking is like there's some tasks that you need to do where you want to like.

08:20.90
Chris Morrell
Right.

08:36.84
Tim MacDonald
Run a migration. Okay I'll run a ced up. Okay, so that's fine. We can define a cedar and we can deploy our application. We can ss a chin and do php artisan Dbc whatever and run some cedar to populate stuff but you might only want to run that cedar. Ah.

08:50.44
Chris Morrell
Um, do you do that? Do you do you see it in production I've never run a seat I don't really use seeers ever. So but I've certainly never run a cedar in production.

08:59.57
Tim MacDonald
Sure sure I have worked on applications where that was the process that we kind of used the cedar kind of helped us. Um you know reproduce and test before we did it. You know, just running an Artisan command or something like that where you're tinkering or something. Um.

09:05.45
Chris Morrell
Um.

09:09.31
Chris Morrell
Sure.

09:16.66
Tim MacDonald
So yup, definitely definitely done the cedar approach but like it's not any different to just doing it in a migration at that point it's kind of the same thing. Um I guess the difference is with Migrations you can blow them away if you use the the Db dump stuff.

09:23.16
Chris Morrell
Right.

09:33.16
Tim MacDonald
Whereas you see are kind of will stay in your codebase. Yeah, so um, okay, okay, so why? what? my idea is is Larryville envoy and kind of augmenting the functionality that already exists there. So so that you can have commands that only run once.

09:33.28
Chris Morrell
Right? Yeah, that's a good point. Yeah.

09:50.29
Chris Morrell
Okay.

09:52.91
Tim MacDonald
On a server and commands that run on one server once commands that run on all servers once maybe commit. Well there's already commands that run on every single deployment. That's just kind of what Larryval envoy is um.

10:06.29
Chris Morrell
Can we can you I have a vague I've never used Envoy I think of envoy as like a ansible or a puppeteer kind of um or a puppet I Guess like ah, a deployment.. The. Tool is that like I guess I imagine that you'd run it locally to connect to remote machines am I thinking about it differently or incorrectly look. Yeah.

10:24.53
Tim MacDonald
Um, yeah, so.

10:29.75
Tim MacDonald
No, that is that is that is correct. Yep what you're thinking about is correct so you kind of have envoy local. You have your your kind of envoy script. It's a blade file. So it's envoy Dot Blade php in your repository and you specify.

10:42.56
Chris Morrell
Okay.

10:47.00
Tim MacDonald
Server so you can say my web server is this ah ip address my worker server is this ah ip address can be. You know an array of those um and then you can write different tasks. So 1 task might be caching the config or caching the routes 1 task might be um, you know restarting the workers.

10:59.78
Chris Morrell
Okay.

11:05.68
Tim MacDonald
Ah, 1 task might be you know, pulling all get down and running the migrations things like that and then you can kind of you can you can write these tasks and then you can also write a story and a story is like a composition of tasks.

11:16.30
Chris Morrell
Okay, and are these like essentially Blade decorations on top top of Bash Scripts or is it running in Ph B or okay, all right.

11:23.41
Tim MacDonald
That is no no, you've you've got it right? Yep so it's it's blades you're running bash scripts so you know you're seeding into a directory running ph prisan queue restart things like that cool. So the thing with Envoy is that it's stateless.

11:36.81
Chris Morrell
Um, okay, all right I'm with you so far.

11:43.36
Tim MacDonald
Um, you know it just runs the same things the same commands every single time. So if you want to restart your queues like you know, just you put that in the story and it goes and restarts the queues. It doesn't know that it's done at it in the last deployment it doesn't know that it has wants to do it every time or whatever um ends.

11:45.55
Chris Morrell
Sure yes.

11:55.89
Chris Morrell
Sure.

12:02.18
Tim MacDonald
The reason that I've come to envoy is that I don't think all problem or I feel like this is somewhat of a general solution again when it when it suits to quite a few problems that are all related. Um, which is you know.

12:12.64
Chris Morrell
Um, sure. Yeah.

12:19.40
Tim MacDonald
When when do you run that cedar. How do you get that cedar to only run once on a machine. Um, but so it's like it's how do you make change to state once.

12:22.67
Chris Morrell
Right? right.

12:34.12
Chris Morrell
Um, right right? and all sorts of different types of changes right? yeah.

12:39.60
Tim MacDonald
Exactly So there was a update we did um on Laravelle Forge where we updated the dependency and the environment variables that it needed had to change.

12:55.28
Tim MacDonald
And without those environment variables. So it was like um, ah you know we'll just call it larall underscore foo and that was what it used to be and now the new environment variable is Larelle Laravell Underscore bar whatever I'm great with examples. Um, and.

13:07.95
Chris Morrell
Sure.

13:13.92
Tim MacDonald
And what happens is if you haven't got that environment variable changed the framework can't actually boot which is kind of similar to if you haven't made the change in the database.

13:20.68
Chris Morrell
Okay, okay.

13:28.19
Tim MacDonald
The application can't actually run I feel like there's kind of a similarity between those and so what I was thinking is can we kind of come up with a solution here where I can say you know hey envoy this cedar should have been run on every.

13:28.33
Chris Morrell
Sure Yeah I can see that. Yeah.

13:46.47
Tim MacDonald
Every deployed environment just once whether it's to like seed my admin roles or something like that and so I could create a task in envoy that's like seed admin roles and then I tell envoy you know do this thing once per machine.

13:52.20
Chris Morrell
Sure that.

14:04.13
Tim MacDonald
Or well it wouldn't know this would be just once per because it's a shared database of course shared state there. Yeah per per deployment I guess per environment. Um and then what it could do is run that thing and now this is again where it starts to break down for serverless but just create a ah have a directory of files.

14:08.44
Chris Morrell
Um, pure system. Sure.

14:23.83
Tim MacDonald
That are just flags to indicate what tasks have been run for that particular machine and that's why you'd want to put that would live on your server itself rather than locally on your machine.

14:35.41
Chris Morrell
And I could see pretty easily having some sort of driver concept where maybe the default is is a file system-based thing but you could do just like migrations. You know you could have a database driver and you know this task ran on this device or this machine you know.

14:50.19
Tim MacDonald
Um, yes.

14:53.92
Tim MacDonald
Yeah, and the reason that I kind of was thinking more fast. Yes, absolutely thing. The reason I was thinking file system-based is that you know what? if the task is to like update the credentials for the database like it has to do that before it can work out whether or not it's run that task before.

14:55.17
Chris Morrell
And this environment that.

15:06.35
Chris Morrell
Right.

15:12.33
Tim MacDonald
And so that's where I think that kind of force falls apart but it would you know it would still be possible if you just never change those credentials or had a dedicated system for this shared data somewhere.

15:23.86
Chris Morrell
Right? Yeah I mean and you're always going to. You're always going to run into that because what if the ah you know what? if the task is you know swap from like a local file system to like a block store mounted file system or something right? You could go the other way and make the same argument arguably.

15:34.87
Tim MacDonald
Yes, yes, yes, definitely and then what I thought would be cool. Is you know we don't know want to be writing bash like sometimes we because I feel like there's my my thinking on this and I think you you kind of message me about this as well like what about. Um, someone saying maybe a different migrations directory or like I had had an idea for like larall actions and I know laraville actions is like I think that's a thing but you know the idea that it's like the same as migrations but for just other stuff and it like saves it in the database and it's just like a Php file you can do that.

16:04.85
Chris Morrell
Yep.

16:14.65
Tim MacDonald
And so then I was thinking well envoy could you could do bash scripts or you could point it to a ph if it's ends in dot php. Then it can just be a Php file and if it's can't if it's just like if you just want to use Php and not write bash then that's fine.

16:30.64
Chris Morrell
Sure.

16:33.45
Tim MacDonald
But maybe the the framework can't boot for this particular thing you still just want to write php for it. Um, then you know having a Php file wouldn't mean that the application would boot. Just be a function you could call like boot or something that would actually then boot the framework. So then you've got access to the framework to be able to do whatever it is you want to do? Um, yeah, this is this was. Again I don't know that this is like all stringing together but in my head I was just having these like wait is this a thing. Maybe this is like a solution to all of this I don't know.

17:05.24
Chris Morrell
Well yeah I mean this is definitely I I see what you were saying in that it's like this is solving the problem at at like a different level right? and and thus it's kind of like solving it's It's a much more generalized solution right? because it's not just about.

17:14.51
Tim MacDonald
M.

17:24.26
Chris Morrell
Data migration anymore. It's about and you know what? what we call orchestration I don't know I I don't know if that's a ah universal term but like just orchestrating the deployment of your system right? and all the different pieces that that go into that I mean right now. Ah.

17:29.50
Tim MacDonald
The.

17:43.75
Chris Morrell
The the thing that I'm a little caught up on is right? we use we use Envoyer right? The lare el product um to essentially do this right? We've got a bunch of different um deployment hooks at different steps in the the deployment process.

17:51.50
Tim MacDonald
Um, guess.

18:03.18
Chris Morrell
And we're running different commands and and most of them are are either. You know the the standard Artisan Commands you know around caching configs or restarting the queue. Um and then and running running Migrations. For example, um. And then you know it could but it could just be a bash script or something like that for for example, um, and so like in that story I guess I can imagine saying I guess you still have. Some of those deployment hooks. But you're essentially just using envoy to trigger the envoy story for that like stage is that kind of what you would imagine right.

18:53.81
Tim MacDonald
I imagine that there's there's a a story for each deployment right? Like let's let's think about ah now I haven't used envoy or full disclosure. So just think at Laville forge deployment script. You've got.

19:03.42
Chris Morrell
Yeah.

19:08.47
Tim MacDonald
Cd into you know, timactdonal me and then git pull to like pull down the latest git information you do phprisan migrates. You do php artisan optimize whatever to like cache the routes and cache like those are things I want to run.

19:22.35
Chris Morrell
Sure.

19:25.86
Tim MacDonald
Every single time like that's my deployment story. But then I am I'm writing a Pr and writing a new feature. Actually you know what what I'm doing is I'm augmenting an existing feature I'm changing an existing feature and the existing feature puts information in the cache. And it puts. Ah um, a list of drink bottles in the cache but the structure of the drink bottle data right? before it was like brand and name as like ah as as like a list. Maybe we've changed that to kind of be a more. More complex data structure and name is now title or brand or something like that. Whatever which is change the structure. So the data in the cache.

20:11.52
Chris Morrell
Sure how about we've got. We've got ah a first name and last name column and we've realized that that's fundamentally flawed and we're just going to a name column is that a.

20:23.13
Tim MacDonald
Ah, love it I love it because that's just good in general. Thank you thank you? Yes as I said not good at examples. Um, so yeah, so perfect perfect things so we've got these, we're putting a list of of of. Names and we've got first name last name in the previous iteration and then we realize you know what? Not everyone has a first name and a last name we should just be asking for the name and so if we then say hey cache give us all the people's names our application. Our application locally is going to work because we're just like using the you know the array driver or something locally. So every time we change the structure of what goes into the cache it just like is valid on our system at all times because it's only for that 1 invocation of the app or whatever. But in production. It's still got first name last name so when we deploy this feature. That has changed and is expecting name to come back. We're going to say give us all of those people loop over them and pull out the name and we're going to get an array offset you know name does not exist you know because first name and last name they're in the cache they're like they live in the cache already. So for this one deployment when I deploy this feature.

21:21.82
Chris Morrell
Sure deb.

21:29.58
Tim MacDonald
I want to bust the cache. So for this one like I've just written this feature I've got a pr up and I've got to say hey whoever deploys this can you like do a can you SsHn and run phprisan cache clear.

21:30.89
Chris Morrell
Right.

21:48.70
Tim MacDonald
To make sure you clear the cache after you've deployed this thing becomes this kind of annual step or if there's like a third -party system you know I've got to go and log in and it's like it's separate system but I want that command that I have to run when this Pr ah is deployed to be a part of the codebase itself.

21:48.86
Chris Morrell
Right.

22:03.52
Chris Morrell
Okay, this is start yet. Okay, this is starting to click more for me I see I So I understood where you where you were going but this this is making more sense. It's like this particular deploy needs this.

22:07.15
Tim MacDonald
Um, cool.

22:18.22
Chris Morrell
Thing to run at some point during the deployment story but it only needs to run once it just it's just make sure that the cache is cleared. Yeah yeah, okay.

22:20.28
Tim MacDonald
Yes, only needs to run once I don't want to bust the cache on every deploy just once with with this particular Deploy. So I go into my envoy script and I say hey I've got a new task that's called Bust the cache and maybe I. Yeah, we'll just leave it at that for now won't complicate. It. So I create a new task that's busting the cache and I say run this on one server and only run it once and then that feature.

22:49.63
Chris Morrell
I I.

22:53.65
Tim MacDonald
Now the commands that have to be run to make this feature. Okay are now part of the codebase they're a part of the Pr I love that it gets merged in it gets deployed and this just happens like it it as it Deploys It runs that thing it persists somewhere that it does run that command.

22:59.79
Chris Morrell
Yeah, yeah.

23:13.35
Tim MacDonald
With the code as a code has gone out and then next time we deploy So we spin up a new feature merge it in deploy it. It doesn't bust the cache that time.

23:23.50
Chris Morrell
Right? right? So okay so I was just looking at the Envoy Docs a little bit in the background and let me see if if what I'm picturing lines up with what you're kind of describing right? So in Envoy I See there's ah, there's basically a couple of ah excuse me.

23:28.21
Tim MacDonald
Is.

23:34.74
Tim MacDonald
Yes, Henry.

23:42.12
Chris Morrell
Um, blade directives one is at task and 1 is at story and at task describes a specific individual task and then at story is a collection of tasks that run right? So you might have like a deploy story or an install.

23:45.65
Tim MacDonald
Yeah.

24:01.71
Chris Morrell
That that calls like the update code and install dependencies tasks right? These are the examples in the Docs and what I could see imagining is I don't know if you could use at once because that's that's you know, already part of Blade but in an ideal world right. There would just be another section of your envoy script at the bottom. That's just at once and maybe it takes like ah, an argument you call it by default It only runs once per environment but you could say like you know per machine. Arrow True or something like that to say wants per machine and and maybe by default then when you run a story excuse me. Um it just it just includes um. Or you know it would be even cooler What if you're once referenced a task so you could say at once and then you know in the arguments you take an array you know after arrow. Update code right? or before arrow. Update code. And then any story that calls that task or anything that ends up calling that task right? checks. Okay was this once triggered for its requirements if not run it in the lifecycle either before or after whatever the different hooks are.

25:38.13
Chris Morrell
That is that kind of line up with what you were thinking.

25:39.29
Tim MacDonald
I was definitely the concepts there a hundred percent but kind of that dependency between the task definition and like the command that it should run with I was more thinking that so the task definition itself would specify. Should it be run once so you'd kind of say all right I want to create a task that is bust in the cache and only and this only gets run once and then in the story. You'd say you know ah update the code bust. Ah. Bust the ah install dependencies bust the cache you know migrate the database and that way it's in the story you can kind of see it in in what order it's got to happen because it might have to happen before the code is pulled down. Or it might have to happen after the composer has done its install so having it kind of as a part of the story itself of like what does the next deployment look like here's the order that these things have to happen in and if it's already happened then it just won't run. It's just a no-op and it'll just go to whatever the next item in the story is so.

26:42.23
Chris Morrell
Right? right.

26:47.11
Tim MacDonald
Rather than having a separate once directive. It's like this particular task is just marked as only running once or maybe in the story itself. It's like wrapped in a pseudo like at once and then or it's like bust Cache Colon once.

26:59.27
Chris Morrell
Um, yeah, yeah I would like I feel like I would want something because the the the downside of that is then you know a year later it. It gets to be kind of Fuzzy. What's happening in this deploy story right? um.

27:20.28
Tim MacDonald
Yeahp I hear what you're saying and if if someone if you've deleted it because you could go and delete those things as well. Right? like you could go and delete bust cache. But then if someone yeah like you're saying like twelve months later comes and goes oh I need to bust the cashche I'll create a but bust cache run this once.

27:25.70
Chris Morrell
Sure.

27:34.57
Chris Morrell
Um, yeah, yeah, you need to be really careful about that. Yeah, that's interesting though I I mean I like I like the concept.

27:35.51
Tim MacDonald
But it's already been marked bust caches a task has already been run months so there needs to be some kind of like identifier for yeah yeah, yeah, yeah, mm.

27:53.36
Chris Morrell
I Think okay to to like to step back like immediately I'm just like yes this is this is interesting because now we're solving even more problems and I love that. Especially I mean this podcast is called overengineered. So like this is this is perfect for that right.

28:03.71
Tim MacDonald
Yeah, like that too. Yeah, exactly exactly.

28:13.10
Chris Morrell
On the flip side we are introducing a ton of of new complexity for what had previously been like just run an eloquent query inside of your migration right.

28:30.85
Tim MacDonald
Ah, hurt her.

28:31.70
Chris Morrell
So is this a solution is this a solution that you have to is this a solution to that problem or is this a solution to sort of orchestrating deploys that also happens to potentially have an answer to that. Question as well.

28:52.36
Tim MacDonald
Yeah I mean it's a solution to something I'm just not sure a hundred percent why yeah that didn't but that's why we're talking about it. Um, so yeah, so it's hard because I keep kind of coming back to yeah the migration stuff like if you want to.

28:57.79
Chris Morrell
Ah I Love it.

29:08.10
Tim MacDonald
So let's let's talk about like the concrete example of populating the different roles on permissions right? because that feels like a good one that we can grasp is like is you want to create the table and then you want to fill it out. So let's.

29:14.18
Chris Morrell
Um, sure.

29:23.59
Tim MacDonald
Let's walk through those steps of what this looks like I'm sorry if we're repeating stuff from the last episode here I did listen to make sure that I wasn't kind of incepted by their ideas. Um, but yeah, the you know what? you you'd create a migration um create a table that's rolls and then.

29:26.57
Chris Morrell
Now go for it.

29:43.60
Tim MacDonald
Probably underneath the kind of sket you go Schema Column Col table create the roles table and then maybe outside on the next line. You'd say you know Roll Colin Col create many and then you'd kind of pass big long list cause you know in. In this migration file you know that that table doesn't yet exist Sorry the data doesn't yet exist none of the roles and stuff because you've just created the table hit me.

30:06.98
Chris Morrell
Yes, but the the problem is you in practice. You typically have to also this is this is kind of one of the gotchas and we didn't really talk about this last episode I don't think but 1 of the gotchas is you really only want that to run.

30:17.64
Tim MacDonald
Are.

30:26.89
Chris Morrell
In production or you don't want it to run during unit tests typically um so so yes, but you usually would have that wrapped in some sort of if statement. It might just be if app if not app running unit tests.

30:33.63
Tim MacDonald
Um, ah yeah.

30:40.51
Tim MacDonald
Um, yeah.

30:46.77
Chris Morrell
I'm trying to remember exactly but that was because that was how this all came up maybe a year or a year and a half ago was we ran into one of those gotchas having to do with testing. Um, but yeah, that's just an aside just kind of popped in my head there.

30:58.51
Tim MacDonald
Yeah, cool. Yeah yeah, no, that's that's a great point one hundred percent. Um, and so that you know we can deploy that that works. Everyone's Happy. So So where's the problem. Um. Guess the problem is you might do a Db dump is is it Schema Dump I think it's schema dump and that blows away all the migration Files and just gives you a dump of the schema. To quickly be able to like regenerate the database without running through each file individually. It's just like here's what you need take it as is that's right? So now when you run Phb Artisan Migrate There is no.

31:32.36
Chris Morrell
Right? So then you've lost all that. So.

31:40.31
Tim MacDonald
You know if we're not in unit tests then loop over these and create like that code doesn't even exist in the repository anymore because those Files have been deleted and so I think that's where the problem starts to come in. Is there other places where this problem can't where like the problem arises.

31:49.25
Chris Morrell
Um, right? Yeah there's there's there's that yeah, there's that the the unit testing problem actually is really just if you do this a handful of times then. You know if you're like us. We've got 20000 lines of of feature tests like ah for features that have nothing to do with that particular data set now you're um, you know inserting. Ah, bunch of records and then removing a bunch of records on every test that that hits the the um refresh database. Um, so that was another issue. Yeah I think there's a philosophical issue of just like this feels a little gross right.

32:31.20
Tim MacDonald
Um, a.

32:38.69
Tim MacDonald
Yep, yeah.

32:38.92
Chris Morrell
Like this migration system seems to be meant for 1 thing and now we're doing something else in it and it just feels a little dirty but um, you know to 1 of the one of the follow ups. So I got I got a couple of really interesting.

32:48.67
Tim MacDonald
Um, and.

32:55.87
Chris Morrell
Ah, responses on on mastodon and on Twitter and and 1 one of them was basically just like just use migrations because just use migrations you know, um that was Brendan white and he you know his his basic point was like kind of to the same same.

33:06.82
Tim MacDonald
Um, yeah, yeah.

33:14.89
Chris Morrell
Point that you were making in the beginning you want something committed to the repository that represents this work. Um, and so that's what we use migrations for is to have sort of a chronological list of the stuff that needs to happen to the database.

33:16.94
Tim MacDonald
The the.

33:34.42
Tim MacDonald
Yep.

33:35.93
Chris Morrell
And so maybe I'm mischaracterizing characterizing his his response because it was it was pretty detailed and ah I'll add a link in the show notes or something but um, you know and I do really appreciate that it's like yes, maybe it feels a little dirty but also. This is a great system that already exists you can get around a lot of the issues and and that's what got me so I had messaged you and said the the one. So another response I got on mastodon was um, Lucas Heller said what they do.

33:56.71
Tim MacDonald
Totally.

34:10.74
Chris Morrell
Is the the database migrator you can pass it a directory parameter right? So by default it looks in a specific directory. Um or it looks in all the registered Migrations directories. So like.

34:16.49
Tim MacDonald
Um, the the who are are.

34:24.14
Chris Morrell
There's the default Migrations directory and then if you use something like modular like it might you know something else might register register other Migrations directories but the miggrator has those directories that it runs through but you can pass it a directory until it just use this directory instead and so his basic take was. If you have like schema migrations that you want to run and then at a separate point in your deploy like your automated deploy process you have data migrations you want to run create another directory for the data migrations.

35:00.49
Tim MacDonald
Um, let's go.

35:00.57
Chris Morrell
And then just have that run as a separate step in your deployment script and I I think that's really clever I think it's it's it really is interesting and um, honestly, that may be the thing that I'm leaning towards as.

35:09.00
Tim MacDonald
Um, yeah.

35:20.19
Chris Morrell
As the like cleanest solution to this problem and then in there if you have something that's tens of millions of rows that need to be processed and they can be done asynchronously I don't have a big problem with spawning some big. Batch job job. Um, from that migration. But then you have that single point of did this get run exactly once for this for this environment. You know it doesn't answer the question of um.

35:42.83
Tim MacDonald
The.

35:50.24
Tim MacDonald
Um, yes.

35:55.16
Chris Morrell
If something needs to be run on each machine right? So if you have I don't know if you have some sort of file system type or you know if you have different Redis caches per per app server or something like that and you need to do something specific to them.

35:58.67
Tim MacDonald
Um, yeah.

36:15.60
Chris Morrell
This would not solve that problem where yours would um.

36:17.90
Tim MacDonald
yeah yeah I I mean I love both of these solutions right? like I love the first one which is just like well just use the thing that exists like I'm all about that 100% and the idea of having.

36:22.36
Chris Morrell
Yeah.

36:33.27
Tim MacDonald
Different directory and then just still using Ph Pris and Migrate is like that is that is very cool I Really like that idea. Yeah yeah, and then that's quite nice for like a a new developer as well right? because you know like.

36:40.56
Chris Morrell
Um, it's super clever. Yeah yeah I mean.

36:49.40
Tim MacDonald
The the roles and permissions thing like that's let let's keep like leaning on. That example, a new developer comes along and you've blown away the schema and like but everyone everyone ran the migration so they've all got you know. Admin and super admin and moderator and they've got all the roles in their database. But this Newde developer comes in runs the migrations and it's like I don't have any roles in my database. It's like ah just run the actions because they don't um'm calling them actions. Whatever the other migration directory is they don't get blown away like they're just permanently there. So It's like a.

37:16.23
Chris Morrell
Yep yep yeah.

37:21.84
Tim MacDonald
How how you set things up I guess I guess it is problematic if Schemas change. Um, you'd have to then update that as well. So there's there's that kind of dependency there but it's still very interesting.

37:28.99
Chris Morrell
Um, yeah.

37:35.66
Tim MacDonald
It's good for production. It's good for production and getting that data kind of moved from a to b.

37:37.87
Chris Morrell
Right? And if you are in a situation where you have a team where folks are coming on and off like and you're bumping up to this against ah bumping up against this a lot. You could probably add some sort of thing That's just like you know.

37:50.64
Tim MacDonald
Um, ah.

37:54.92
Chris Morrell
It reads the schema and only runs that data migration if the schema is like the shape that it expects right? and then if it it you know if at some point in the future. It gets to a place. It's not ideal though. It's it.

38:02.81
Tim MacDonald
Guest. Yep.

38:11.81
Chris Morrell
Yeah, the the whole issue of of um, doing the schema dump throws another wrinkle into it that I hadn't really thought about yet. Let me let me give you the 2 other there are 2 other options that came out. Let's let's just put them all on the table because this is interesting. So.

38:21.97
Tim MacDonald
Um, something that's um, yeah, yeah, hit me placed. Yeah.

38:30.56
Chris Morrell
Another another response that I got on Twitter Ed ah ed grosspinner ah said he just has a run once command and that command just gets run at each deploy but it it basically just takes the MD 5

38:41.92
Tim MacDonald
Um, this is.

38:50.32
Chris Morrell
Of itself and just aborts if it's run that that hash before so you can either store that hash on the file system or you can store it in the database I Love this I think this is so clever and so it's just like if you change the run once commands.

38:56.54
Tim MacDonald
Yeah.

39:01.93
Tim MacDonald
Um, yeah, that's interesting.

39:08.91
Chris Morrell
Handle Method. It runs but then the next time it runs that hash has been executed so it just ah, ignores it and I think that's like I think it's probably not the right solution in a bunch of cases. But I think for the right. Developer or small team of Developer. It's just it's beautiful I think it's It's a very clever solution.

39:32.80
Tim MacDonald
I really like that idea and this this has reminded me of and I'll talk about it after we've kind of gone through the other solutions but Taylor had a tweet about this kind of concept as well and it reminds me of that implementation I think. 1 thing that would trip me up is like someone runs phpcs fixer and like change as a rule and now the mdfive has changed and you didn't want that thing to change and now it has but but I still like it. It feels like in the same kind of headspace that I was in around like there's like a marker to be like this thing is run. You know whether it's like a.

39:52.17
Chris Morrell
Yes, yes yeah.

40:06.16
Chris Morrell
Yeah, yeah, yeah.

40:09.71
Tim MacDonald
You know and just an empty file is a flag or like the Md five hash of their whole file. Yeah no I dig that I dig that.

40:15.14
Chris Morrell
Yeah, and then the last the last solution that we came to at the end of the the last episode is this is very confusing because of the the larevell actions package that um is is pretty popular but there is um.

40:28.86
Tim MacDonald
He.

40:32.39
Chris Morrell
A package called that's like laraville actions by the dragon code or the dragon code actions Dragon code actions. Um or actions by the dragon code for Laravell is the official title I guess and.

40:41.58
Tim MacDonald
Okay, that's amazing name for something either way.

40:51.10
Chris Morrell
Right? It's such an um.

40:51.96
Tim MacDonald
That's a great title I Just want to. It's so good. What a good package name. Oh my gosh I Love that.

40:56.83
Chris Morrell
But it's It's a little unfortunate you know, just from from a name confusion perspective because Lairville actions is is so popular but um, it's essentially what you had kind of talked about at the very beginning it is.

41:03.37
Tim MacDonald
Ah, absolutely yeah, yeah.

41:13.87
Chris Morrell
Migrations but for anything right? It gets they get the the weather they have run gets stored in the database just like migrations but it can be you know any type of thing and they can be scoped to an environment so you might have like an action. That's just you know.

41:15.41
Tim MacDonald
I Hope the.

41:32.39
Chris Morrell
At and in internationalche we have ah you know Pgpr is in Dev setup command that does a bunch of stuff if you have a new Dev environment including you know, pull a bunch of sample data and and set up the npm packages and you know all that stuff right.

41:37.42
Tim MacDonald
Um, this is.

41:51.37
Chris Morrell
Um, and you know effectively we could replace that with something like this. That's just a bunch of actions that only run in the local environment. You know, um and this is this is probably the other one that's most appealing to me because it's it's.

41:51.52
Tim MacDonald
Um, the.

42:10.39
Chris Morrell
It's essentially all the things that we like about migrations. But it's just ah, another space that's dedicated to things that don't feel right for Migrations and are a little bit more broad than Migrations and it you know it kind of gets into what.

42:21.64
Tim MacDonald
Are the.

42:29.89
Chris Morrell
What the envoy solution is where it's like you can run anything you know, um, but because it's because it's within the context of an Artisan command it. It doesn't have some of the advantages.

42:37.14
Tim MacDonald
Um, and.

42:49.20
Chris Morrell
In that you know if you need to change out the database connection or something like that. You're already inside of a booted app. It's going to be problematic in a different way. Um, but that starts to her right right? Sure Yeah yeah.

42:58.56
Tim MacDonald
And that's assuming the app can actually boot as well. Yeah, yeah, yeah.

43:09.90
Chris Morrell
But yeah, those are those are just the other things that came up and I and I think there's a lot to be said for all of them. Um I Love this I Love how much there is in just this one little tiny question right.

43:12.84
Tim MacDonald
Um, who.

43:16.96
Tim MacDonald
Yeah, definitely. Yeah, yes, it's great. Um I didn't realize all this stuff existed. So I'm I'm love it up. Um, yeah so Taylor also kind of had a solution to this which um, he tweeted about. Quite some time ago which was again it feels. It feels very similar to to all these things where it's I don't know what the command was but I believe it was essentially an artisan command and the artisan command just had a list of classes and we'll just call the classes lack actions right? so. Whatever it is. It's you know one of the one of them is populate roles and um, the class itself was just an invocable object. So. It's just essentially just a function wrapped up in a class and when you called when you kind of invoked the class it was and.

43:53.73
Chris Morrell
Um.

44:12.57
Tim MacDonald
I Don't know how to say this word. It's one of those words that I've read everywhere and then when I say it out loud I I'm always like I didn't say Yes, thank you now I don't even have to say it. That's so I need to record that and then I can play it every time I Want to say it? Ah yeah, yeah, gosh. Yeah, so that.

44:16.23
Chris Morrell
Um, item potent. Um I damn potent.

44:31.55
Tim MacDonald
Ah, you know they were all responsible for working out whether or not they should run themselves So the roles and permissions thing might be like hey do all of these roles exist in the database if they do just return like don't do anything if they don't you know, maybe insert the ones that don't yet exist and that would allow you to.

44:34.60
Chris Morrell
Oh okay, okay.

44:43.45
Chris Morrell
Right.

44:50.82
Tim MacDonald
Update that when you add a new role that has to exist you know if you had admin and moderator and then six months later you want to add a super admin you just go in there and say you know add super admin to the list of roles. It goes do all 3 of these exist. Yes, they do don't do anything. If They don't all right work out which ones don't exist and just insert that one. Um, which is definitely an interesting approach as well and then all you do is end up with a really nice, clean list of just you know classes that are just objects and you're kind of in that same in that same. World except that The class itself is responsible for working out whether or not should run.

45:29.72
Chris Morrell
Um, right right? Yeah I like that approach I mean I like that approach I Wish you know I I wish I think what I like about.

45:32.44
Tim MacDonald
Are the.

45:44.18
Chris Morrell
Some of the other things that we've talked about is there's a formality to them right? where you're you're using sort of ah maybe it's not a core convention. But you're using some sort of convention that's like prescribed by the package whereas.

45:49.21
Tim MacDonald
Um, ah.

45:59.67
Tim MacDonald
Yes.

46:03.20
Chris Morrell
Something like this you know because we can come up with our own homegrown hack together solution for anything right? This this feels like and I mean quite honestly a lot of times That's maybe.

46:10.40
Tim MacDonald
Oh oh yeah.

46:19.37
Chris Morrell
Better solution for things and we and maybe we reach for like more sophisticated solutions too often. So there's there's an argument for that too. Um, but yeah, it just feels like if there was a loose.

46:37.92
Chris Morrell
Standard around this type of thing whether it's something like what you're proposing or whether it's this dragon actions thing or whether it's like ah a formal like um like a formal flag.

46:56.13
Tim MacDonald
If.

46:57.75
Chris Morrell
In the miggrator that tells like that you can do Ppran migrate and then you can do Pgp Artisan Migrate Dash dash data or something like that and that's like a so special thing or dash dash after and dash dash before to be like these are the things that run before Migrations These are the thing that run.

47:06.29
Tim MacDonald
Is the.

47:17.74
Chris Morrell
After migrations I like I would love to see there be sort of a standard way. That's not standard just to 1 team but's like sort of standard to how you approach it in Larel. But that said you know we've got. We've got a bunch of great options here and. For the most part everyone gets gets along just fine with with what we've got so um, but I like that I like that that idea of just a bunch of classes that are just really self-contained responsible for figuring out.

47:40.64
Tim MacDonald
Oh yeah, definitely.

47:52.71
Chris Morrell
You know do I need to run and then if so this is what I do.

47:56.00
Tim MacDonald
Yeahp Yeah, exactly it's It's quite nice and and I guess in a way that's kind of influenced my thinking on the envoy thing as well where it's just like it's just a list of tasks to run and the tasks themselves again. They can be responsible for working out if they have to run or you can use Envoy's kind of once command. To like you know it'll it'll remember if it's Run. You know it's It's almost like you know, composing your solution out of all these ideas I think is is definitely probably most of the time the way to go right? because there's so much context, everyone's going to need for their Apps. So.

48:22.89
Chris Morrell
Right? right? for sure. Yeah I mean I think probably your system like the number of servers you're deploying to and the size of your team is going to change change this pretty significantly for sure.

48:37.12
Tim MacDonald
Yeah, if you're on serverless. Um.

48:39.19
Chris Morrell
If you're deploying to 1 or yeah or if you're on serverless for sure. Yeah, you know if you're if you've got 30 app servers to deploy to or one app server to deploy to that's going to change the story or 0 right? Or if if you are going serverless.

48:53.85
Tim MacDonald
Definitely definitely? Yeah, yeah, so there was there was something else I didn't actually realize was kind of in this world that I Prd to the framework that we just like I kind of like.

48:58.35
Chris Morrell
Yeah, amazing. Okay.

49:10.80
Tim MacDonald
Lost motivation for it. But I thought it was really nice to begin with and now I'm kind of coming back around to it after having this conversation which is the idea of having um I called it like warm you know like warming your application. So imagine um, a request comes in and ah.

49:22.18
Chris Morrell
Okay.

49:30.55
Tim MacDonald
The first time you hit this endpoint. You need to go and like download a text file with a few ip addresses put it in the cache and then the next time a request comes in it just can like pull it out of the cache. Well the first person that hit that had to pay the price of downloading those ip addresses from a text file off the internet right? so.

49:39.74
Chris Morrell
Sure.

49:47.64
Chris Morrell
Sure.

49:49.56
Tim MacDonald
So that very first visitor has to pay the price of warming the cache whereas if you could do that as a part of your deployment story. Um, then no one has to pay the price of the cache or everyone kind of pays the price in the downtime if it's part of downtime or maybe you can not. Maybe you can essentially go yes if the first request comes in they have to pay the price but maybe no one will hit it by the time you know we've actually already warmed the K Shop so what I kind of envisioned was ah you know in the kernel in the in the console kernel there's a kernel ph p.

50:13.72
Chris Morrell
Right? right.

50:23.53
Tim MacDonald
In our applications in our framework application. Um, there's a schedule function and the schedule function allows you to kind of define jobs to run ah on the cron on the cron as a cron job essentially so you can say every 5 minutes you know, um.

50:25.85
Chris Morrell
Yep.

50:42.49
Tim MacDonald
Go and do this thing every day. Update this particular thing or send this send this web request to make sure that I know my server is still kind of running a cron or whatever I I imagined ah a new function under that called Warm which would essentially tie to a php Artisan Warm command.

50:50.70
Chris Morrell
Sure.

51:02.90
Tim MacDonald
And in there you could kind of warm your application so you could say all right? Go download this file and put it in the cache and so you do Php Artisan Migrate Php Artisan Cache warm and it would warm the cache or warm your application. Whatever that would be for your particular application. And so so yeah, so you'd migrate the database you'd run this command that command would then populate go and fetch that file put it in the cache that when that request comes in it doesn't have to go on downloaded itself and and then you know there's probably ways you could then. Implement some of these things we're talking about where you could say you know once on the end of it like fluently call you know run this thing and then do it once and if you do that then it you know persists that it's run. It's done it somewhere and then it's kind of not so much this huge new concept. It's just like the scheduler.

51:44.36
Chris Morrell
Right.

51:57.40
Tim MacDonald
Um, you know in a way that the migration aspect is just like the migrations. It's like quite nice that it's a similar concept. Okay.

52:02.66
Chris Morrell
I Love this I Okay because I because I'm thinking I mean let's let's let's overengineer this like. Why Why just have a warm option right? We've got the scheduler right? that that says um, you know, basically put this command in line to be run when some other command gets.

52:20.94
Tim MacDonald
Plays boom.

52:40.41
Chris Morrell
Executed right? that's ah What's effectively what's happening is like tell the scheduler to run this command daily at noon or whatever. Um, but really all that's saying is like put this in a list somewhere and then when I run Pgpe Artisan Schedule run or whatever. The command is I can't even you know.

52:41.95
Tim MacDonald
Yes. The.

52:59.74
Chris Morrell
You set that up once and then you never look at again. So I can't think of what it is um but all all I'm okay, all that's doing is saying like here's a bunch of commands that have been queued up to run which ones should be run right now right.

53:01.78
Tim MacDonald
It grows. Yeah I think it is scheduled run. Yeah.

53:17.24
Tim MacDonald
Um, the.

53:17.93
Chris Morrell
So what? if ah, there was like that same concept but just generic where I can essentially say put this task in this bucket with these constraints.

53:37.63
Chris Morrell
And the scheduler is just one. You're it's almost. It's almost a step. It's an abstraction above the scheduler where the scheduler would could be an implementation of this same thing right? Um, where its job.

53:49.89
Tim MacDonald
Um, with a roof.

53:53.84
Chris Morrell
Is to run all the commands that have been scheduled. Um, but you could also have a warm bucket right? That's for cash warming or you could have a like pre-deploy button.

53:55.53
Tim MacDonald
Um, yes.

54:09.70
Chris Morrell
Bucket or a Post-dep Deployy bucket or a pre-migration bucket or a post-migration bucket. You know like you could just have any number of buckets as you need it right? And so so now Pp art is in schedule Run just becomes an alias for like Ph P artisan.

54:14.85
Tim MacDonald
Um, this.

54:27.81
Chris Morrell
Ah, Command I don't know what this would be called but like ah ah, bucket run and then the next argument is like Cron or something you know like that. That's basically just one of the one of the buckets right? and then you could have then you could.

54:37.73
Tim MacDonald
Then.

54:44.67
Tim MacDonald
Um, then.

54:45.91
Chris Morrell
Provide the nice convenience features on top of that like once or you know on on one server you know the things that the scheduler does you could pull the ones that are useful outside of the scheduling concept. Um into.

54:53.77
Tim MacDonald
Um, the.

55:03.78
Chris Morrell
The the more abstract system but that's kind of an interesting idea.

55:11.93
Tim MacDonald
Yeah, ah yeah now now I'm getting aside in the about of the get how funny yeah I think 100% like you know, obviously the scheduler only run can only run per minute.

55:16.30
Chris Morrell
I Mean ah.

55:28.11
Tim MacDonald
Um, and but we want to run at a specific instance. We don't want to have to wait to the end of the minute for crime to like trigger this thing but yeah, being able to just be like these are the things that you run when we're when we're warming or when we're pre-deloying or whatever which you know again, we can We can achieve this through articles and commands that are just in our deploy script or.

55:33.64
Chris Morrell
Right.

55:47.41
Tim MacDonald
Or whatever you know event listeners that like listen for the migration started or migration ended and do stuff but I like the formality of having it in the kernel and the idea that it's the same concepts and like you said like being able to say you know maybe the warm commands receives like a ah task ah builder. Whatever and you can say all right task run and then you say you know whatever the task is that you have to do you know download this file or maybe it probably accepts a closure right? and you know downloads a file puts it in the cache and you can say you know on 1 server and it's the same api as the scheduler where it's on 1 server and that's.

56:21.82
Chris Morrell
Right? Exactly the out.

56:25.88
Tim MacDonald
That's quite nice again that that there's no new Concepts and you get to use the existing concepts.

56:31.29
Chris Morrell
Um, I mean frankly, it could be. It could be the schedule object right? What if I want this this task to run daily right? like.

56:42.20
Tim MacDonald
Literally.

56:45.21
Chris Morrell
If I deploy 6 times in the day I only want it to run once but if I when I deploy if I run it you know noon on on Thursday and then I do 6 deploys but then I deploy at one p M on Friday I wanted that command to run again.

56:53.59
Tim MacDonald
Interesting.

57:01.78
Tim MacDonald
Right? So you're you're saying you're saying that you want to be able to like to to have the ability to say all right run this command on 1 machine once per day when I deploy.

57:04.30
Chris Morrell
Right.

57:20.71
Chris Morrell
Right? right? daily daily now becomes like just to an additional constraint. It's like run it but only run it once daily at most right or or maybe it's once monthly or you know, whatever, whatever and and.

57:21.37
Tim MacDonald
If I deploy interesting. Okay, that's utristic.

57:38.65
Chris Morrell
You'd have to you'd have to split it out right? because some things wouldn't make sense right? like daily at right is not going to work in this context because you don't know what time Deploys are going to run at um so this the current sort of schedule.

57:38.81
Tim MacDonald
E.

57:44.91
Tim MacDonald
Yeah, yes. A great. Yeah.

57:56.49
Chris Morrell
Object would need to be a subclass of some parent object. That's that's got less configuration but you could potentially move in more than yeah, you know you've got without overlapping or then ping or. Like you've got a bunch of things that would make sense. Um in these more abstract kind of contexts so then you essentially instead of just having 1 scheduler that runs with pgpr is in schedule run.

58:16.98
Tim MacDonald
The.

58:26.92
Tim MacDonald
The.

58:33.84
Chris Morrell
You would essentially have as many schedules as you wanted and I might run Ppr is in schedule run every minute on aran and then I might run Ppr is in schedule run predeploy.

58:41.60
Tim MacDonald
To.

58:49.57
Chris Morrell
And run that every time I deploy you know what? I mean like.

58:50.56
Tim MacDonald
Yeah I like the idea of having it there or co-located. That's that's quite nice.

58:59.84
Chris Morrell
And and it's the right place. It feels like the right place to put them right? because that's where we already um define all of these commands that need to run at different times in our app like it kind of.

59:03.94
Tim MacDonald
Um.

59:15.96
Chris Morrell
Even if I didn't know where to go that might be a place that I'd start start just to see what happened I love this and then um, you know those right.

59:20.58
Tim MacDonald
Yeah, yeah, definitely yeah I dig this as well. This is cool to overeering now.

59:35.00
Chris Morrell
Those commands could essentially do what Taylor's solution was you know in that command it could just be you know before I execute I check to see and if if there's nothing for me to do I just ah I just abort right? and you can.

59:37.94
Tim MacDonald
Um, yes.

59:49.36
Tim MacDonald
Yep yep.

59:52.70
Chris Morrell
If you have some task that's potentially going to take 4 hours to run. You can use process control to spawn off a background process. You know you you can you can or you can push things to the queue or you know in your command you can manage the resource needs in a bunch of different ways. But.

01:00:04.19
Tim MacDonald
Um, yeah.

01:00:12.13
Chris Morrell
It's still the same single story of where you where and how you define things.

01:00:15.13
Tim MacDonald
Um, yeah, yeah I think that's nice. Um.

01:00:22.30
Chris Morrell
Interesting. Ah I love that that we came to a solution that no one can use because it doesn't exist yet.

01:00:29.14
Tim MacDonald
Yeah, yeah, a hundred percent that's that's the way to do it for sure because then we have to build it so that's nice if.

01:00:38.91
Chris Morrell
Um I I don't know maybe um, maybe I'm over maybe I'm over simple or I don't know it seems it seems really.. It seems like a really good. Um. Seems like a really good sort of like synthesis of all the different things that we've talked about that uses Laravell's existing way of doing things really nicely.

01:01:05.38
Tim MacDonald
Does it actually so it sounds good in a vacuum does it actually so like coming back to the practical like problems that we're trying to solve like what would that look like in code. So you know we've got to so writing a feature that needs to. Bust the cache then in the I'd go into the kernel.php the console slash kernel dot php I'd go into the we'll call it the warm function for for now. What it' a warm function receives the task. Maybe it is just all in the schedule. Maybe not. But. Just give it a separate function for now just to kind of keep it separate and I'd say task once and then whatever the command is to bust the cache. You know, php artisan cache clear. Um and because I've tagged it with once.

01:01:44.30
Chris Morrell
Um, sure.

01:02:00.78
Tim MacDonald
And you know not I don't care about other deploys. It's just like just this one time I want you to run this I guess we still have that problem of um, you need some kind of identifier so that you can you know that it's this particular iteration of the deployment that we're we're going with.

01:02:10.48
Chris Morrell
Right.

01:02:17.97
Chris Morrell
I Mean maybe once just takes a string you know.

01:02:19.56
Tim MacDonald
Um, yeah, yeah, yup, actually that's that's a good good point. Maybe yeah yep, some kind of identifier. Um, and that was then we can merge that in it gets deployed the warm command gets invoked when we deploy. It busts the cache. Everyone's happy next time it Deploys It doesn't happen. Everyone's happy so that works.

01:02:44.21
Chris Morrell
Yeah I mean you know to your point the rolls and permissions thing right now now we're going. We're creating say 2 migrations one for the roles table 1 for the permissions table then we're creating a new command.

01:02:53.40
Tim MacDonald
Um, yeah.

01:03:02.48
Chris Morrell
Or maybe we're creating a cedar that seeds the rolls and permissions and then we're going in the kernel and we're adding to the Post-deploy hook run this cedar.

01:03:05.21
Tim MacDonald
Um, the.

01:03:19.87
Tim MacDonald
Um, yeah, once yeah.

01:03:22.20
Chris Morrell
Once yeah, it is I mean it's a little less um directly connected for sure.

01:03:36.65
Tim MacDonald
Um, yeah, there's a yeah some separations there.

01:03:40.22
Chris Morrell
You know, which which I think any of these solutions that we've kind of talked about do have that fundamental problem of like when you introduce Abstractions things are more abstract right? That's going to happen.

01:03:53.61
Tim MacDonald
A.

01:04:00.56
Chris Morrell
I like I like it a lot for a lot of the things that you have to do as part of a deploy you know, um, like for example, ah right? we use sidecar to um.

01:04:08.65
Tim MacDonald
O.

01:04:18.45
Chris Morrell
Deploy Lambda functions for um, you know, rendering pdfs ah with with puppeteer. Um and right now when you know when we went all in on Sidecar we had. Add a couple of different things to the deployment process right? There's um, there's a there's a there's a command that that has to run as part of our deploy to push the new versions of those sidecar functions up arguably.

01:04:50.39
Tim MacDonald
The.

01:04:55.60
Chris Morrell
In this world I could have just added like we already have a post-eploy hook um set up that's just php art is in whatever to run this this new thing that we're we're describing and I could have then just gone into the code and said like just put this. Sidecar Command in that bucket to run and then I wouldn't have to like mess with our actual deployment process to add a new command to the deploy which is nice I do like that and I think for the thing that you're describing like the caching.

01:05:20.80
Tim MacDonald
Yes.

01:05:30.25
Chris Morrell
Some some sort of Okay, we just want to. We just want to grab some some data file like we you know we we we do this with um with a list of like all of the official. What are they Called. Routing numbers for bank for Bank Accounts. You know we we just pull that list of routing numbers in the bank name. Um, so that we can offer them for autocomplete and in a couple of different places and validate them so like something like that right? just just make sure that at least every once in a while those routing numbers get pulled down.

01:05:50.10
Tim MacDonald
Yeah.

01:06:06.26
Tim MacDonald
And you probably got something like that in the schedule right? like once a day go and like download those. But if you're spinning up a new environment then maybe that you know midnight isn't for 7 hours and that new environment doesn't have that.

01:06:08.44
Chris Morrell
Um.

01:06:13.21
Chris Morrell
Um, yeah, yeah.

01:06:24.91
Tim MacDonald
Thing I mean you're probably hooking it up to a shared cache that doesn't really matter. But yeah, yeah.

01:06:26.69
Chris Morrell
Um, right right? Yeah yeah I mean I think that maybe we stumbled upon a great solution for another problem.

01:06:39.14
Tim MacDonald
Ah.

01:06:41.30
Tim MacDonald
Ah, again, it's definitely a solution I'm just not sure what to yet? Yeah, exactly exactly.

01:06:42.57
Chris Morrell
Ah, man. Yeah, yeah, yeah, we've got a hammer man. We just got to find some nails.

01:06:53.82
Chris Morrell
Ah I This has been great. This has been really fun. We've been going for a little over an hour I could go I could keep going but I and I do want to just like put out there right? There are.

01:06:55.44
Tim MacDonald
Agreed okay, cool. Oh man. Yeah, we could go Dave I'm sure.

01:07:10.83
Chris Morrell
There are a bunch of there are a bunch of these and I Love if you have any of them that you want to share for for future episodes I'd be curious like one sort of sub topic of this or I think it's It's really an entirely is a topic on its own but like. Another one that I really want to come back to is referencing specific records that exist in the database in code because that comes up all the time I know that like this vendor this specific vendor exists.

01:07:37.40
Tim MacDonald
Um, in yeah yeah, in.

01:07:46.23
Chris Morrell
And I need to do some special thing and there's a whole.. There's a whole feature in the system. That's all about you know grabbing data from their proprietary system and pull it in but it needs to reference that vendor and and it's like do I hardcode that id. Do I use like some sort of like lookup table all the time. There are a bunch of different questions about how to approach that How do I do it in testing How do I make sure you know because an obvious solution is just seed all the vendors kind of that's why it touches to this. It's like.

01:08:13.61
Tim MacDonald
Earth.

01:08:24.91
Chris Morrell
Just seat all the vendors that you know you need um with maybe known ids and then set the auto increment to a thousand so that the first thousand ids are reserved. There's like ways to do it. But then for all the other features in my app I've seeded 80

01:08:27.80
Tim MacDonald
Um, the.

01:08:35.93
Tim MacDonald
Yeah.

01:08:43.36
Chris Morrell
Vendors into the database that I'm never going to need for that feature test or something right? you hit those same problems. Um, and there's a bunch of stuff around that um and like there. Ah we've we've we've we've approached it in a bunch of different interesting ways I think the the one that.

01:08:45.80
Tim MacDonald
Little Even yeah.

01:09:03.14
Chris Morrell
Seems I don't think it's I don't think it's the Silver bullet I Don't think it always answers it but sort of like inverting that where you actually store like the the fully qualified class name of the thing in the database so that you.

01:09:18.00
Tim MacDonald
Are.

01:09:21.64
Chris Morrell
You are well I'll just leave it. It's that um, another one is another huge one that I think I this this I don't know if this comes up for you all the time but status transitions.

01:09:25.10
Tim MacDonald
I Love it.

01:09:39.20
Chris Morrell
And timestamps associated with them right? I have a model that might go from like new to assigned to scheduled to completed and then maybe ah to like some issue status and then back to new because like. Something happened where the whole process need to start over and I want I want to have a string in the database that is the status of that thing because I'm going to be querying for assigned assigned jobs all the time I don't want to be.

01:10:01.45
Tim MacDonald
Um, event.

01:10:15.87
Chris Morrell
Querying against some pivot table I don't want to be doing some weird complex query on a bunch of Timestamp Columns I want just a a column that's called status so that I can do the queries that I'm going to need to do 90% of the time but I also the.

01:10:22.74
Tim MacDonald
Um, yeah.

01:10:26.46
Tim MacDonald
And it can be duplicated data all the way down, but it's like it's so much better to just see that value there if you're looking at the database as well. It's like oh yeah.

01:10:33.57
Chris Morrell
Yeah, yeah, exactly. But at the same time I'm going to want like time stamps associated with all those transitions and a lot of times those timestamps are also going to be fundamental. It's like. Well I want to know if this was scheduled when was it scheduled at if this was completed when was it completed at um and I think that there are a bunch of there are a bunch of different ways to approach this and I have yet to find one that feels like this is the thing that I'm going to reach for first.

01:10:49.76
Tim MacDonald
The.

01:11:05.28
Tim MacDonald
Is.

01:11:06.97
Chris Morrell
I Think that all the solutions that we've come across at least are things where all of them kind of have bad tradeoffs and I don't I'm not confident that I know which one I think has the least bad tradeoff. Um, another one is custom. Oh go ahead.

01:11:19.73
Tim MacDonald
Yeah, and I think in that in that particular problem I think the thing that and you know now I'm going to underedine engineer stuff because I haven't thought about this at all, but the the idea that like you know, ah you you ever gonna have to. Query against the updated date of the status and if you're not it doesn't have to be a dedicated column right? like you never going to update it. You're never going to query against it or if you are querying against it is it something like you're manually querying against so you don't really care about performance in which case you know whack it in a Json column.

01:11:49.24
Chris Morrell
Right.

01:11:51.30
Tim MacDonald
And then just have a a status updated at or status. You know that could even be ah, an array so that you can like keep the timestamps and like see the you know now you've now you're doing event sourcing an adjacent column in a database. Maybe that's not a good idea I don't know.

01:12:04.27
Chris Morrell
Um, no I I have been down that path ah another one that I have on my list is custom relations and I don't I don't know if this is a thing I Wonder if this is only a thing that I ever deal with.

01:12:07.26
Tim MacDonald
Um, yeah I bet. Yeah.

01:12:21.46
Chris Morrell
But we deal with customer relations all the time and part of that has to do with with we've got a bunch of legacy data. That's not modeled in a way that Laravell's eloquent really likes to work with and so in a lot of cases.

01:12:33.18
Tim MacDonald
This is.

01:12:39.12
Chris Morrell
We'll just deal with the inefficiencies and it's fine, but in some cases um the data is is used so heavily that we need to query it in really performant ways. Um, and so there will be a lot of like these. These very very specific custom relationships that have to do weird Joins or unions or Joins and unions against multiple columns and you know have to handle the eager and non-eager scenario and I just I'm more just like. Is this something that other people deal with and how do other people approach these these any data that's related in not just oh, this is a belongs to or has many you know? um.

01:13:28.27
Tim MacDonald
Ah I want to hear this episode I'm nowhere near smart enough to even contribute to the episode but may I want to hear this episode the the fanciest stuff I get done with eloquent is like install and want to. Jonas is it statomy Jonas Stademir packages that's that's as fancy as I get with eloquent that you know there's some. He's got some mad packages out there. Yeah.

01:13:43.30
Chris Morrell
Um, yes, yes, those yes are incredible and we do the you know the limited like ah being able to add limits to your has Manys and stuff like that like there's just oh.

01:13:57.52
Tim MacDonald
Yes, Yes, exactly.

01:14:00.44
Chris Morrell
Just go grab one of his packages install it and then I can just add limit to the relationship and it just works and under the hood. It's doing all these wild like table or what are the window functions and stuff that I have no idea how they work. But.

01:14:07.21
Tim MacDonald
Yeah, yeah, um.

01:14:16.56
Tim MacDonald
Exactly Yeah, exactly do they just work. They're so good. So good.

01:14:16.90
Chris Morrell
We've never had a problem. So ah yeah, relations in Laravele is my like my little pet peeve I I feel like I have in my head sort of like.

01:14:26.62
Tim MacDonald
Is it.

01:14:34.13
Chris Morrell
A way that I think relationships could work that would make building complex relations a lot easier but but because ah I don't know I can see why they kind of. Evolved in the way that they do but now it would take pretty much a whole overhaul of that system which obviously no one wants wants to see happen. Um, but I just bump into relations all the time but I'm wondering. Do you have any of these that you're like man I bump into this. And it's always like yeah we can make it work but ah.

01:15:13.55
Tim MacDonald
I'm I'm sure that they are right like we all bump into things like that and then you know you bump into it again at some you know later down the line I don't think I haven't got anything off the top of my head but I'm definitely going to start like writing them down as I come up and I'm going to be sending them to you so that you can solve them for me.

01:15:29.70
Chris Morrell
Amazing I Love it I Love it? Well this has been so much fun. Um, and and now we're going to have to like build a whole thing. So I love that too I'm going to be thinking about this.

01:15:30.88
Tim MacDonald
Yeah I agree.

01:15:44.22
Tim MacDonald
Definitely.

01:15:45.38
Chris Morrell
And maybe I you know I was wondering. Is there going to be ah a third episode on this topic because that was the other thing like I kind of envisioned this idea where we have these like mini seasons of of 2 or 3 episodes and I and I was I was kind of thinking I don't know if there's going to be a third episode but maybe there will be. We'll see after this.

01:16:01.51
Tim MacDonald
Um, yeah, yeah, would love to hear it.

01:16:05.11
Chris Morrell
Gonna be ah, but um, yeah I don't know Thanks for thanks for coming on. This has been great and I hope I Hope we do it again.

01:16:10.85
Tim MacDonald
Yeah, thanks for having me. It's been fun. Yeah please I can I can over engineer anything you throw at me. That's that's my bread and butter I got this? yeah.

01:16:21.85
Chris Morrell
I Love it all right? Good night.

01:16:26.72
Tim MacDonald
Catch you later.