Over Engineered

00:04.85
Chris Morrell
All right welcome to over engineered the podcast where we ask the very important question. What's the absolute best way to do stuff that doesn't really matter so a while ago I had an idea to do this show. Basically there are things that we run across as like. Developers that every time it's like well it doesn't matter just kind of we'll just do the thing that gets it done this time and every time it feels like well it doesn't matter. We'll just do the thing that that gets us there this time but it keeps on bugging me like someday I want to come back to this and so I decided. The heck we're going to get people together. We're going to argue about these these little tiny details until either. We're really sick of arguing or we have come to some sort of useful conclusion. So ah, there's a bunch of different topics. We're either going to do an episode. Maybe. 2 or 3 episodes with different people on the same topic until we kind of come to some sort of consensus and then we'll move on and ah if that doesn't work. We'll try something different. So the first well actually before I get into the topic. Ah. Today I have my friends Skyler and Bogden with me. We are all ah Lairrivelle pachp developers at Internatchi um, and we are going to talk about something that came up recently.

01:34.59
Chris Morrell
Um, but before we get into the details you guys want to say hi.

01:38.29
Skyler Katz
Um, hi I'm Skyler.

01:39.32
Bogdan Kharchenko
Hey guys I'm Bogdan I'm here with the internet team I currently live in South Carolina let's do it.

01:45.69
Chris Morrell
Nice.

01:47.80
Skyler Katz
Oh man I guess ah yeah, I'm in St Louis

01:52.47
Chris Morrell
Ah, all right? So here is the setup. This is something that I feel like comes up over and over again and it's just like you've got a new feature right? So let's let's use this. We set out a nice little example in the the docs so I might as well look at it right. So you're setting up a new ah like roles and permissions feature and so we've got a migration for the roles table. We've got a migration for the permissions table and we know from the start that there are going to be a few specific roles like. There's going to be a guest There's going to be an admin. Maybe a staff a couple other roles right? and there's going to be like 15 different permissions and so the question always comes up like what do you do? do you throw the actual insertion of data in your migration. Do you set up a cedar and run that cedar after in production after you've done the migration. Do you set up an artisan command that's really just like a one off command for inserting that specific data or is there some other approach and so like I was looking back even. Ah. You know over a year ago. This topic had come up and and I went on Twitter and I posted a little poll. Oh another option is just hey go into Tinker or go into your like you know, sql editor and just do the inserts directly. Um, and.

03:25.40
Chris Morrell
There was a little bit of consensus but it was kind of all over the place and so ah before we kind of talk about the pros and cons like gut reaction which is it.

03:39.81
Bogdan Kharchenko
Um, which which is the best option is that what I kind of yeah I mean yeah I mean I will say like I've done the manual insert into the database and it's certainly not it. You know there's certainly a lot of tradeoffs for the speed. Um, you know.

03:42.86
Chris Morrell
Yeah.

03:57.58
Bogdan Kharchenko
I Feel like there's opportunities ah to use. Ah you know, like ah constants and you know just reference php strings you know or just strings in general. They reference a database and have some sort of command that just takes some sort of class full of constants and. Syncs them into various places whether that's roles or Permissions. So I feel like those are like a pretty good starting point that I feel like most people reach for.

04:28.66
Chris Morrell
What about you Skyler.

04:30.86
Skyler Katz
Yeah I mean I I want something as easy as as Ph P R is in Migrate but it always feels kind of gross to put it put it in the migration. Um, especially if you're you know, like doing that Migrate. You end up making a schema from your migration Files and then suddenly you've got this weird stuff in there that that blows up but you know bespoke commands every single time feel off to.

04:57.23
Chris Morrell
Yeah I mean that's that's one hundred percent where I am is like I hate the idea of having to SsH into one of the app servers after a migration and run a command and then yeah have a command that's sitting around that. Really doesn't do anything. It was really just a one off thing. Um, but at the same time I can acknowledge that every time I've done this where I've thrown the data in the migration I've felt pretty dirty afterwards like I know that that that doesn't.

05:34.85
Skyler Katz
Yeah.

05:34.94
Chris Morrell
It doesn't feel right either? Um, and and I also feel like 1 of the times I don't know if it's the last time but 1 of the times this came up the other issue was now when you run those migrations in unit tests. Ah, all this data is in there. So if you've written tests that kind of assume like if you're if you're doing migration-based testing where you're where you don't have some sort of you know world that you seed ahead of time but you're just factoring up the data that you need for each test now. You've run into a circumstance where. You've got a bunch of records in your database that you weren't necessarily expecting to be there in your tests and and they could get in the way if you make certain assumptions. Um, which is definitely another downside of running them in the test in the migration rather.

06:26.78
Skyler Katz
Yeah I mean I've done them I've done things where you know you're you're writing and writing tests and you are factoring up data. But then you still expect like your permissions table to have those permissions like even in my test I don't I don't want to like factory up the admin role I just wanted to like have it there. Um, but there are other times where you've added a new column to a table and you need to ensure some some data is there and you don't want that in your test. So yeah, we we need some solution that's going to work for for. Kind of all across the different environment levels that you have.

07:04.19
Chris Morrell
Right? Well and I mean I guess we could take it a step back. Take a step back and say what about circumstances where you have to manipulate existing data right? because I don't have much of a problem doing that in my in Migrations right? if I. For example I don't know why I would do this. But if I were switching from say a if I if I had a given name and a family name column and I wanted to add a just switch to a name Column. Um, or maybe or or maybe a better example would be like I have a given name and a family name column and I want to add a display name. Column I don't have much of a problem in my migration saying you know, update Users set full name equals can cat given family Name. You know like. That feels okay to me does that am I are you are you guys on board with that at least or does that feel dirty and migrations too.

08:06.45
Skyler Katz
Um I think like it's not. It's not the worst but it's still it still doesn't feel. It feels like a little bit like you're just abusing the migrations. The fact that Migrations run once and that.

08:20.44
Chris Morrell
Right.

08:25.36
Skyler Katz
Is it feels like we're abusing that feature of migrations.

08:32.78
Chris Morrell
It's also it also really depends on how much data there is right because if we're talking about a table with you know 100000000 records in it I'm going to approach that very differently than a table that has you know 5000 records in it.

08:45.45
Bogdan Kharchenko
Yeah I will say you know I feel like the lairville migration system is really for doing column changes and creating tables and adding indexes. It's not really designed to be inserting. Data or manipulating data in my mind at least there's probably I'm sort I'm sure there's a way for us. You know to do these steps but you know I feel like if we take a step back and think about well why are we trying to automate this right? kind of um. You know why are we trying to add roles and permissions at the very beginning of the or at the ah you know the time where we created the roles and permission stable. We need to see some data and I feel as if you know that is just an entirely different step. Um, you know there. It's 1 thing to migrate data and there's another thing to actually seed that data and even in the example because that you mentioned in regards to the given name and the family name to combine it into a name It's not really a migration. It's kind of like a data manipulation. Um. Mechanism you know chances are if you were ever to run that. Ah, you know you probably would not be dropping a column immediately. You would run that command in the background and you know ah have that data populate and then there's probably subsequent code that goes out or maybe it's behind a feature flag.

10:16.63
Bogdan Kharchenko
Ah, that actually takes that new manipulated data into effect you know and I feel like ah those commands are like very specific utility Commands. You know you don't just pipe them in. You know it's It's just over Time. You're. Data Model has changed through migrations and over time to accommodate those changes you need to change the data. But when you're migrating a fresh set of data for testing none of that took Place. You know what? I mean all, you're doing is bringing the world up. Up to the latest schema changes and you know I feel like I have to agree with you guys adding some sort of seeding mechanisms inside the migration system is just you know a very dirty trick that I feel like we've been. Ah, used to because that's just the way Forge Deploys or envoy or you know and all these other services. They just kind of run this by default and I feel like we should step back and say like well perhaps our deployment steps should um you know run some sort of commands. Um, you know in addition to the migrations. Um, even if they're like you know, manually configured.

11:31.32
Chris Morrell
I mean that's a good point this this is this is all in the context of a like cicd strategy right? If you're not doing continuous delivery if you're not doing some sort of automated deployments. This is less of a concern because if you're a whole. Deployment strategy is already pretty manual doing 1 other manual task you know is not that big of a deal on the flipside you know at least the way we run things we do. We can do ten fifteen deployments in a given day. And some of those will be. You know, just queued as soon as the test pass they auto emergge and and get deployed and so like yeah, you're absolutely right I am spoiled by having this really lovely. You know github actions and and envoy or workflow that just. Makes everything happen automatically. Um, and and I don't know now that I'm spoiled I don't want to go go back from that I want some other automated process that does this piece for me and.

12:40.61
Skyler Katz
Well I think too like I've come on to projects before where like the database like someone just hands me a database file and they're like here's the local database. Or you know you run a command to import a local database I've also come to projects where you just run the migrate command in larevell and then like maybe there's a cedar or you just have nothing and you just have to register for an account and then they're like oh yeah, the permissions table go into nova locally and just like add in. Add in this Google spreadsheet of permissions. Um, and and I feel like those are also things that you'd want to solve for with this approach is like how do you yes in your tests like you're testing the latest version of things but like the everyday. Getting things set up or you're spinning up a new staging environment like how do you get the data where you need it to be and and that's I think the other use case for some sort of like item potent data manipulation.

13:49.40
Chris Morrell
Right? Yeah I mean we definitely underutilize cedars in you know in our project. That's not something that we really heavily lean on and I can see I can see. You know a database seear or running different database seeers being part of the solution here. Um, but you know they they kind of come they feel like I don't know database seeers don't feel like they're meant to be used in production in laravell at least? um. You know? And in fact, when you try to do Dbc. You get a big warning if you are in prod um, but you're right, a lot of the cases where you would want this in production right? like these are the 15 15 permissions and 5 roles that are going to be in the app by default from the beginning. You're also going to often want them in. You know your local dev environment or a staging environment or ah, you know, depending on the feature something like. Like permissions and roles. That's so central to an app. You're probably going to want that in a lot of your tasks. You know, whereas some other feature that is really just ah, a ancillary feature. You probably don't want to seed.

15:13.41
Chris Morrell
That data for all your tests because you only need it for a handful of tests in in the whole suite. So it also depends on what type of data you have. But.

15:19.89
Skyler Katz
Yeah, seems like there's maybe what 2 different types of data that we're talking about. We're talking about data that we need to be in the system. Ah that. You could run over and over again the command and it's going to make sure that data is there. But then there's there's the other portion which is massaging existing data into a new format and those when you're massaging data into a different format like probably don't want those to run in a test or you really wouldn't like. It wouldn't help us whereas running something to make sure that data is there in the database I don't know if I'm don't know if I'm making sense there.

16:00.48
Bogdan Kharchenko
I yeah I totally agree I feel like you know Skyler is 100 % correct in this It's you know I think the the case for rolls and permission is quite clear because those are like very heavily dependent on like you know checks like is this user a. Guest string or something whereas you know there's like these other long commands or utility commands some may say um to actually manipulate the data. So I think like what we're talking about is this initial use case of the roles and permissions like how do we. Kind of make these small changes during the migration process or some sort of post-deploy process. Um, which are you know may not be like as tricky as you know running some sort of long process that. You know manipulates a lot of data Would you agree with that.

16:55.91
Chris Morrell
Um, I mean yeah, it it almost makes me think that. Yeah, we almost want something that's like we're kind of like migrations in that you know, arguably once you've run your migration. You don't need it. That file to sit around but it's definitely useful for it to be there. Um, and certainly you know if you're going to set up another environment. You do want it to be there. But in the day day-to-day operation of the application. It's it's kind of just like a excess legacy file right. And in the same way it it almost feels like what we want is either 1 off commands that are perhaps tied to a specific migration or are in an entirely separate.

17:35.25
Skyler Katz
Are.

17:51.90
Chris Morrell
Like are represented in the database in a separately an entirely separate table where um, you can run some second command that says hey these are the these are the data data migrations or whatever that. You haven't run yet would you like to run them. You know.

18:12.14
Bogdan Kharchenko
Yeah I totally agree I feel like I kind of understand what you're getting onto I I do worry. So let's maybe let's step back and think about this right? So what you're kind of suggesting perhaps in my mind is um, you know we create some sort of um you know. Data migration status table of some sort and we create ah files or classes that represent what needs to happen and you know during deploy we kind of say like hey you know we run the migration workflow you know as per usual. But then there's another command that is run right after that and we would just take you know, um the files that we've defined and you know effectively run a database seat or in production and then just record that that thing happen. Um, the only thing I would wear is that is that true Chris is that kind of what you're leaning towards the guy.

19:12.88
Chris Morrell
Yeah, something like that I mean I can't help but think you know there are a lot of other commands that are sort of like 1 off that it might not be terrible like if you know. This this podcast is called Overengineered. So I'm going to let myself go down this path of um, you know how many how many times have we written artisan commands that are really just there for a single use and is there some.

19:49.15
Chris Morrell
You know, kind of migration type approach where it's like you're not actually introducing a whole new command. But instead you're just introducing another thing that can be run by some sort of generic one-off command runner. And you could kind of do that. Um, then you could do that for both cases right? It's like these are all things that need to be run once and only once right, um, unless you know what you're doing but by default they're they're only run once and so you just drop. Some you know php file that's got sort of ah ah, a date time component in the in the file name just like migrations do um and there's ah ah you know there's like a database record called one off commands or whatever you know and it just. Gans that directory for files it compares it against the one off commands database and just like the miggrator it just loops through them and runs the commands 1 by 1 and that way you know in the the deployment. Story we would just run that after migrations had been run.

21:08.25
Skyler Katz
And I think you could even do something where you have like either a either a use a trait or just a public public property where you're like ah like run in test or is testable. So then. You know, similar to refresh database. You can have like refresh data that's going to run but you might have a 1 off thing that you don't want to have run in your tests or you have some other thing like your permissions where you want that to run in your test suite but like maybe by default and then of them run. Unless you you know use in tests trait or something.

21:50.35
Bogdan Kharchenko
So so one of the other options or on the same topic right? like I just looked in the background. Um, how onvoyer deploys data right? The you know the files everything and you know we run things like you know, um. Obviously ppr is a migrate queue restart and you know this could be effectively a second command if I'm understanding you. Ah Chris um, you know that we just pipe at the end of the deployment process after the. Um, releases activated effectively right? We just basically say ah deploy this and then run this command and in the case of the test world. Um, you know you may not you know you could choose whether or not you want to run that command. Um.

22:23.21
Chris Morrell
Yep yep.

22:39.99
Bogdan Kharchenko
Similar to similarly to how we do like you know database refreshers you may choose to additionally run this data cing command in your test and I think that's a pretty decent solution. You know to start with.

22:57.53
Chris Morrell
Um, yeah, so I actually I was looking through this Twitter thread from from last April or whatever and I saw someone responded with this ah this package called laravelle actions which is a little confusing because there's the the. The the big larabel actions package. Um, that is you know for for you know, ah single actions that you can you can execute from different contexts. Um. But this is this is essentially what we're talking about. So it's dragon code actions or dragon code larabel actions and it looks like essentially essentially they're doing kind of what I described right? You have these actions that are. Very similar to migrations. Um, except that they are meant to to do something like if you look at if you look at where where you know actions. So this is how they describe it actions are like version control. For your actions process allowing your team to modify and share the application's actionable schema if you've ever had to tell a teammate to manually perform any action on a production server. You've come across an issue that actions resolves so that I mean that does sound like maybe someone has.

24:27.29
Chris Morrell
Has come up with a ah, pretty good approach to this I don't I hate I hate that it's called actions just because um, if you are using what is the other one's called Laraville actions right? Um, yeah.

24:40.54
Bogdan Kharchenko
Yes, yeah, I'm looking at that package and you know to be honest, you know it kind of sounds like what we were talking about. There is effectively a ah database table that represents all of the actions that have ran. And it. It seems like you can roll back actions although I am curious when is it intended to be ran I I haven't figured that out just yet. But this looks pretty cool.

24:58.85
Chris Morrell
Yeah.

25:08.85
Chris Morrell
Um, well I would imagine It's the same thing right? we could we can um you would just put it in your your deployment pipeline wherever it made sense if you wanted to.

25:20.25
Skyler Katz
Yeah, and they have these like you can protect it once false or true whether whether it runs every time you run all your actions or you know execution in a specific environment takes an array or string of like which environments to ah to run it In. Or excluding from be interesting to which is kind of yeah which is a little bit like what I was what I was mentioning of like how do you run this in the run this in testing but not elsewhere in this.

25:40.30
Bogdan Kharchenko
Oh interesting.

25:40.45
Chris Morrell
Yeah, that's neat so you could you could have an action that's like Dev only or something like that.

25:57.68
Skyler Katz
Kind of resolves that. Um, yeah, this is it's interesting. The logos fun. Yeah.

26:05.64
Chris Morrell
This dragon.

26:07.40
Bogdan Kharchenko
Yes, So okay, so I mean it sounds like we've come to a quick solution but you know what are the drawbacks to to this approach right? Um, you know I feel like ah you know is there. You know how do we mitigate issues in production If This runs will this you know, hold up Deploys and so on and so Forth. So.

26:30.92
Chris Morrell
Um, right? I mean the the biggest concern that I can think of is um, you know what happens in the case of an action that that's you know, operating on some million number of millions of records right? like. If we were I don't know wanting to do some some really big manipulation. You know, ah maybe like segmenting the audit log into like yearly yearly segments or something like that right and we wanted some. Some command that could take ah all the audit records and and generate like an audits 2022 and audits 2021 and and you know some I'm just trying to imagine something that's like dealing with millions and millions and millions of records. Um.

27:26.65
Chris Morrell
You know in that case yeah is that likely to that's that's going to hold up deployment for sure and I guess it kind of depends On. There are some actions that you can kind of run whenever and it doesn't matter and there are other ones that you want to have run before the deployment finishes right.

27:45.20
Skyler Katz
I mean maybe there's some way to like Mark an action is cuable. It's just like it's a thing that has to happen once but it doesn't have to this package doesn't do that but it would be. You know that's one solution is just like this action is cubable and. And needs to run 1 time but it doesn't it doesn't affect the use of the deploy versus like well you kind of need your permission stable set up before you can utilize.

28:09.50
Chris Morrell
Right.

28:17.29
Chris Morrell
Um, right? Yeah, you could just you could I mean you could just queue you could just have your action be queuing a job quee an anonymous job. That's just a one off thing or a batch of a bunch of anonymous jobs or something like that. Yeah.

28:30.54
Skyler Katz
Yeah.

28:33.34
Bogdan Kharchenko
So That's a good question I mean Ah, so basically you would still need to create some sort of file right Previously. We talked about running we would you know create some sort of command where you would either add it to your. Deployment Pipeline as a thing to run after Deploy or you would have to like shell into your server and run that command so you know you would still need to create a place to house more complex logic right? So that is a benefit. Um. You know with this package solution here and.

29:08.72
Chris Morrell
Um, yeah, you would you would basically you would still have you would have a file for each action right? and you would you would.

29:17.37
Bogdan Kharchenko
So.

29:25.67
Chris Morrell
Just put in your um, you know in the envoy or config or whatever you would put like Php artisan actions. Whatever you know actions install or actions. Yeah actions run or whatever I can't remember.

29:36.20
Bogdan Kharchenko
Sure that.

29:41.52
Chris Morrell
Although there is also kind of it's Interesting. There's ah, there's a concept of before actions. Um, Also so you could kind of split you could have something like in your deployment program. Scripts You could say like run the before actions and then run migrations and then run all other actions. Um, that's kind of neat.

30:06.56
Bogdan Kharchenko
Right? I do see the example that they have is basically you know you would do migrate migrate and then actions before and then you know you would do your additional steps and then restart your queue and then run the subsequent actions at the very end so it kind of.

30:22.62
Chris Morrell
Right.

30:25.36
Bogdan Kharchenko
Talks You know is effectively the same thing we talked about earlier.

30:28.47
Skyler Katz
It's like your your before actions or like database your after actions might be code running running queued jobs or doing something that needs access to the whole codebase.

30:31.12
Chris Morrell
Right? so.

30:40.84
Chris Morrell
Right.

30:42.16
Bogdan Kharchenko
And with that fresh data right? That was just migrated. Okay, so.

30:45.75
Skyler Katz
Thing that thing that I always wonder about in these things is like you don't want to end up with a situation where maybe the data exists in the database. But maybe it doesn't and how do you. Is it up to the developer to just like write first or create ah on that permission or is there a way to like in these I don't know some sort of like exists method that you can write some query yourself.

31:07.84
Chris Morrell
Right.

31:24.35
Skyler Katz
If It returns True. It just like doesn't it just like bails out some sort of like validation that run that like more than I don't know more than just an if statement in the in the action handler just like some sort of validate. That this should still run.

31:46.40
Chris Morrell
Well this this makes me think of this is kind of getting into episode 2 ah, where because this is this is a whole other class of question that is definitely related but it's it's ah it's a whole other topic of you know I need this. This record to exist my code expects this record to exist I need to reference this record in some way and like how how do I make that happen and we could jump into it now. Maybe let's hold off for a minute minut minute minut minute more and see where we where we land but because that is a whole other question and I could. I could see basically just saying like ah you reserve maybe maybe you reserve before actions for things that need to do data manipulations that are necessary for your application like. For the next state of your application and then you reserve other actions for any other sort of like hey this needs to run but the app can run while it's while it's happening. But um I mean this this makes me think so.

32:52.35
Skyler Katz
Um.

32:59.30
Chris Morrell
You know we were experimenting with um with a smaller side project. Um, and and we were playing with some of the new hot technology and so for a little while. We were experimenting with like deploying to fly and using planet scale for the database and planet scale basically forces this same question on you which is or I don't know. There's not a great story for automated automated. Automated migrations with planet scale because they have their whole sort of version of of pull requests. But for ah, database deployments database migrations. Um, and it's it's a little bit different but essentially. You know their their point is there are some migrations that you need to do before the application changes and there are some migrations that you need to do after the application changes in some cases in in the original case that I was giving if I were switching from a you know. First name and last name column to just a name column I would need to copy migrate to and add a name column I'd need to then copy my data from the first and last name columns to my name column and then I'd later have to migrate to drop the first and last name columns.

34:29.90
Chris Morrell
But before I did that I'd have to to deploy a new version of the application that uses the the new new attributes. So it's sort of like a multi step process and depending on the. Data and schema manipulations that you're doing sometimes they need to happen before deploy and sometimes they need to happen after and while they are 100% right? that that is technically true in my experience. 99% of the time you can just run migrations during deploy and as long as you know that that's happening and you've architected the code correctly, it just works and you never have to think about it and for those few cases where. It does really matter. Well then you just deploy the migration in one phase and then you deploy ah the the code that relies on that migration in a second phase but I don't know in in the last several years I can only think of maybe having to do that once or twice it just. Don't find that it comes up a ton and so it's it's kind of the same here. It's like well hypothetically you're going to need to sort of orchestrate these actions and migrations and deploys and in this like really really.

35:55.80
Chris Morrell
Sophisticated balancing act. But in reality I don't know 90% of the time you can just migrate and then run a command that that updates you know a few thousand records in a second and. Just live with the consequences of oh maybe someone happened to hit hit the app during that second but realistically the way we do deploys. That's not even usually going to be an issue. So.

36:25.24
Bogdan Kharchenko
I Think the question is not whether or not it's performant or when we do it. It's like I think is the migration system the right platform for making this change because yeah, you're right? You know we could just stuff everything you know we could just easily. Ah say like hey when this thing runs you know, just execute this file. So I don't think it's really an issue with just deploying I think like you know, just getting data in there. We obviously found solutions whether it's. You know some other utility putting into a migration file I think is just kind of like well how can we do it efficiently you know because um, that's part of the challenge here in my mind.

37:13.15
Chris Morrell
Sure yeah I mean I I think I'm convinced that doing it in migrations especially since someone has already done all the work of writing a package that does close to what I want. I'm I'm more easily convinced that doing it in the migrations is not necessary. You know I I am always a little bit hesitant to add. Um, add a new package that that is sort of like a. Foundational decision right? without really thinking about it because you know if we were to go with something like this I think you know we'd want to be going all in with it. So I I want to know that we're going to want to stick with it for the long haul and not just like you know ah have. 2 or 3 actions that we run through it and then never use it again and now it's just another thing that slows down the deploy by a few seconds. Um, but that said I mean I think the model whether it's through this package package or something that you just kind of put together yourself. Model of having whatever they are actions or 1 ne-off 1 ne-off commands or or data migrations or or whatever you wanted to call them having them as their own thing that sort of follow the same general flow as migrations but are not migrations does.

38:46.83
Chris Morrell
Does feel good. Yeah.

38:47.10
Skyler Katz
Yeah, you know I liked I liked this package because they they talk about just invocable methods but but later on they talk about having up and down methods I don't know of how I feel about like migrating data having a rollback strategy like.

39:02.90
Chris Morrell
Yeah, well.

39:03.90
Bogdan Kharchenko
Yeah I will say Taylor.

39:05.89
Skyler Katz
I think I think you just should migrate forward I mean I feel similarly about database migrations. But it's I know like that's a I don't know maybe that's too spicy of a topic.

39:19.50
Bogdan Kharchenko
I will say you know I feel like I listen to a podcast with Taylor and you know he revealed to me many years ago that the down method is optional right? and part of the reason it's optional is because like yeah chances are. Ah, when you know you're going to roll down. It's not as easy as just here just roll back these changes a lot of things probably happen. It's it's not as simple as just roll down you know, ah roll back the migration in production you have to kind of move forward. Almost.

39:54.84
Chris Morrell
And so this I don't know if this is is an appropriate topic for this podcast but I I was having this argument with Aaron Francis ah like a few months ago and he is very staunch ah never never never go back. Ah proponent and I get I get that I don't know I get that thought process but the the 1 time and and I think it's only 1 time. Maybe maybe it's been 2 times. But. Though the 1 time that ph p artisan migrate down saved my butt is worth writing the down methods in every migration to me because every once in a while right. The the thing is when you are in the like worst case scenario we just deployed code that is is like really fundamentally flawed in some way that no one no one anticipated. The only thing like you're in panic mode and you want to just like. Get things back to what worked as quickly as possible and like I said I I can't remember what this scenario was but I know that that there was one of these deploys maybe 2 2 or three years ago where.

41:27.66
Chris Morrell
We just deployed a feature that had some fundamental problem with it and you know it took the whole site down everything was busted and I was able to you know quickly. Ah roll back in Envoy or so just deploy the previous Um, the. Previous ah commit to the production branch and Run Ph P Artisan down and you know within a minute Everything was back to where it needed to be and to me being able to do that. Just you know Now it's just like ah it's like a security blanket for me. It's like I know that I have it and I I do I'll Run. Ah you know, artisan down sometimes ah in in testing certainly in local testing. Um, but I'll run it just to make sure it works right? I'll I'll migrate up and down to make sure that you know if there's a feature that I'm maybe a little bit anxious about like let me just make sure that if I need to I can get down and and. I Don't know I'm still a believer.

42:42.51
Skyler Katz
Yeah I mean I think like all things it has its place I I don't know I've been bitten I've been bitten by it in that you know we we migrated a big table like millions of records adding a column and it. You know in staging it didn't have millions of records and the thing was fast and so we thought like something happened. Let's roll back but the table was locked it rolls back locked even longer and you know like. I think it's all so you know it's all situational but you just end up getting in a situation where you're like oh well shoot rolling back didn't work like what what else needs to happen and ah but you know it's I think there are times where it can be useful and there are times when it can be dangerous.

43:19.82
Chris Morrell
Um, yeah.

43:37.49
Chris Morrell
Yeah, no I can appreciate that. Well.

43:37.92
Skyler Katz
To roll back.

43:39.98
Bogdan Kharchenko
Yeah I will have to second skyyler you know I've been in a similar situation where everything worked fine then the migration time down production and you know it's like well we just made a bunch of code changes things are writing to different tables now because we've changed the migration. And you know like adding an index timed out effectively and it's kind of like well if we roll back. You know, um, you know you're going to lose all of that data that has just been added by users in these new columns. And you know it's ah it's a tough call to just roll back and in my experience I too write the down method and migrations. But it's mostly just so I can migrate down when I need to create a column change in the migrate up. Um. You know a strategy of you know migrating database when I'm doing local testing so you know that's kind of where I reach for the down method I don't think I've ever really ran it in production I think it's super dangerous.

44:43.24
Skyler Katz
I Suppose that's where planet Scale Migration strategy comes into place where they were on both sets at the same time and write data to both places and.

44:54.30
Chris Morrell
Yeah I mean that's incredible, right? Um, okay so this this is this is maybe ah, a silly thing for me to be ah annoyed with but I'm I am a little annoyed that. The other are not annoyed but like it's a consideration. The the other thing that makes me question this specific package is like I have always thought that the ah the the Loris leave. Ah let you know.

45:30.84
Chris Morrell
But I don't know how to say his last name. Um the but the the Lareville actions package that is you know for actions that are um. Just a single unit of of action that then can be you know called from controllers or called and from um commands or called inside of jobs. You know, like the example of um. User registration is an action and you can execute it from your registration controller. But also if you want to manually register a user you can call the same action from a command and um, that whole that that concept and that package has always really. Appealed to me and I've always been curious about introducing it into a project that I'm working on and so there's a part of me that's just like kind of hate the idea of having 2 different things called actions.

46:35.30
Skyler Katz
Sure Well I think Dragon code actions is maybe a stepping stone for what you're looking for what we're looking for. But um, you know.

46:51.70
Chris Morrell
Yeah I guess so I can live with that.

46:51.20
Skyler Katz
And name change is easy enough.

46:54.81
Bogdan Kharchenko
Yeah I will say I've looked into this layer of actions package before I mean yes, it is compelling and I feel like it is interesting and you know I feel like both of them have their own use cases and you know I understand the name conflict. Obviously we can rename it. Um, but at the same time you know you could theoretically use. Could you use this as a migration ah strategy like seeding data as actions is that a thing or no.

47:30.80
Chris Morrell
Which which action are we talking about? Um, well yeah, you could.

47:32.16
Bogdan Kharchenko
delerableactions.com one could it work in the same way because it does seem like it does a handful of things.

47:40.11
Chris Morrell
But yeah it it doesn't but it doesn't have that piece where it tracks which actions have been executed in the current environment and so you'd have to re-implement that but I suppose you could right? if you just added that.

47:46.18
Bogdan Kharchenko
Yeah, ah.

47:53.21
Bogdan Kharchenko
Sure I see.

47:58.54
Chris Morrell
If You essentially used the Lareville actions package the lavilleactions dot com package as the foundation for these action objects and then you essentially just like. You know this This has a whole like as object as Controller as Job. So Maybe you would just have um a special interface that's like as tracked action or something like that you know and if as.

48:23.33
Bogdan Kharchenko
I See sure.

48:32.68
Chris Morrell
And action implements that interface then it would be somehow pulled in as part of this command I don't know. But.

48:44.49
Bogdan Kharchenko
Sure I Do wonder you know if we step back a little bit like why doesn't Lu will have something out of the box right? Then are we like overengineering something that should be already here and you know are we just under utilizing cedars. Um, you know why isn't there something? um. In the framework itself. So.

49:03.62
Chris Morrell
Yeah, that's a good question I I.

49:07.78
Skyler Katz
I mean I think I don't know like you could use cedars you can use database Migrations I think the thing that most people use Anecdotally would be writing commands that that. Change the data when you need to and I've seen I've seen like yeah you write your migration and then you have your you write your command and then you call the command from your from your migration file so that it only runs once but during the deploy but then it's like.

49:45.65
Chris Morrell
Ah, the the worst of both worlds are.

49:46.80
Skyler Katz
Still there to run other times which feels it feels gross, but it's like well how do I how do I get around like the 18 steps to deploy at a large company but still get this thing to run without access to like. Server.

50:05.57
Chris Morrell
Well in the the gigantic sample size of 37 people a year ago a little more than half said they they just make the changes in the migration 30% said they were on a custom command and the other 20% said they ah. They just update directly through tanker through sql of some sort.

50:26.87
Bogdan Kharchenko
Yeah I definitely feel like we need a solution those options all sound bad and that's what we do I feel like there has to be almost like part of the reasons why I feel like layer of all appeals to me and even like things like rails and stuff like that is the Migration system. All these features like queue systems and everything like that and this is one of those areas where there's almost like not a clear picture of how things need to be.. There's a lot of you know, ah homemade solutions to this problem and I feel like you know. Narrowing down a good solution Even if it is this actions package. Um you know, ah just to make that kind of a thing that is available as part of the liberal ecosystem I think that would be an incredible step forward for the framework itself. So.

51:20.88
Chris Morrell
Well, um I I mean I agree and I I I happen to know that somebody ah who is ah who is a lare of el core contributor might be working on something here and I'm hoping that we can. Ah. Can convince him to to hop on for maybe the second episode because he ah he teased something that he he said there may be a little lower level solution to this problem that no one has suggested yet and I've been thinking about that and I have no fricking idea. Ah, what that means. But um, but but until until we get a glimpse of what that might might look like I think that this dragon code action seems like a really great option I think i'm.

51:59.10
Bogdan Kharchenko
Interesting.

52:15.58
Chris Morrell
I'm tentatively willing to say that that's that's my that's my vote for how how we might go about accomplishing this moving forward to.

52:23.25
Skyler Katz
Yeah, you know and my my hypothesis as to why? it's not in the framework already is that like people don't stick around projects long enough to run into the problem of having to migrate large amounts of data like either. Either your side project like it's big. But maybe it's not so big that this is hard to do or like at big companies that are running laraval like developers leave every few years and so different people implement this in different ways and like just nobody. Nobody has the time like no product manager at a big companies can be like yeah, go contribute this back to the framework and figure out like a good way to do it. They're just like no write write the sequel directly or like whatever you know to to make things happen.

53:17.74
Chris Morrell
Yeah, it's also there's a balance like it's always a balancing act of like how much is appropriate to be in the framework and and how much is appropriate to be in. You know, third party packages because you know there's there's always that ah temptation to just. Add everything at everything to lareval and I get needing to push pretty hard against that from the maintainer perspective.

53:42.76
Bogdan Kharchenko
Yeah I could see that yeah I don't know it's just something that kind of ever since we started thinking about this topic. It's something that's been looming in the back of my head like well why isn't there. You know it seems like a pretty you know, approachable solution especially with this layerable actions thing. You know I feel like we're I don't know I suppose when we have the special guest on given that he agrees. Ah maybe we'll get a glimpse of the future ahead of us. So I'm excited.

54:12.65
Chris Morrell
It it could be I mean it is kind of an interesting thought experiment to say what? if what if migrations had some sort of grouping mechanism. Um. And so you could filter down migrations by group and then I could in my migration class I could have some sort of public static um public static property or or public. Static method that could be called. That's like the migration group right? and then in your application you could run like Phb Artisan Migrate Dash Dash Group Equals schema right or default or whatever and then later you could do ppr doesn and migrate um dash dash group equals data or whatever or group equals post deploy and group equals pre-eploy or you know like if you just had some mechanism for you. For um, sort of defining migration groups and and like by default the group is is default and from the command the group is default implicitly unless you set it and that way if you define a migration that's in some other group.

55:46.50
Chris Morrell
It just only gets run if you call the miggrator for that group that could actually solve this problem for the most part using all the same tools that we have in place right now.

56:02.61
Skyler Katz
Yeah, almost like almost like P B unit test annotations like the group grouping your tests you group your migration that or I wonder if there's like some updates to the cedar component itself.

56:09.65
Chris Morrell
Right? right? right? yeah.

56:20.91
Skyler Katz
So like maybe seears can keep track of if they've been run before or they can keep track of so you can seed you can seed or you can like seed once um.

56:34.98
Chris Morrell
Right? That's cool and.

56:34.99
Bogdan Kharchenko
So that is an interesting. That's an interesting um you know idea like what if so-called special guest is kind of implementing thisleural actions or has an idea of doing thislural actions as part of the ceders. You know because I think like 1 of the big downsides of layerable actions is there's now another set of files in you know, perhaps the database folder you know alongside with migrations and cedars and now you have these actions to keep track of and I feel like. That is just like another thing to look out for and like well when did this run and when did that run you know I mean it's it's a bit difficult to ah and I'm not sure if I could answer that question because I haven't used that app. But you know how when do you know when something was run right? um. You know during which deploy like is it connected to a migration um or is it just another. You have to like match up timestamps ah to see you know during which kind of workflow does happen given that you were creating a table and you know adding some records you know that's. Personally my kind of like challenge with I suppose this actions app ah package project.

57:54.49
Chris Morrell
Yeah, yeah, it's just like a whole new thing to keep track of.

57:57.67
Bogdan Kharchenko
Right? Perhaps like a good way to you know to solve this is to use database cers or inside the migrations. There's like a before hook and maybe like an after hook and you know. Ah, during like the phprs and Migrate Step. You know, maybe there's like a before thing that you can then call your little command to do whatever data manipulation and maybe there's an after Command ah method on there that you can just Define whatever needs to happen after. That thing has successfully ran. So Maybe that's also an option.

58:39.47
Chris Morrell
Yeah, or even like I was just thinking like what if you could just have any public function on your miggrator and like artisan migrate is just implicitly calling artisan migrate. Up and you you swap out artisan migrate down with something that is artisan migrate space down like you're giving it the first argument of down and it's just running all the down commands. Ah right? and you could do.

59:12.91
Bogdan Kharchenko
So that's an interesting idea.

59:17.81
Chris Morrell
Artisan Migrate Foo and it just runs through all the migrations. Um and looks for the function called Foo and runs them you know I understand that there are some problems with that. But it's kind of an interesting idea.

59:25.28
Bogdan Kharchenko
So name full. Yeah.

59:31.54
Skyler Katz
Um I don't love that.

59:37.56
Skyler Katz
Yeah I don't I don't love that. But I think it's It's definitely interesting. Um, yeah I don't know.

59:48.31
Chris Morrell
I Feel like we we've come to the ah to the other side of this hill and now we're walking down the path to worse solutions.

59:54.14
Skyler Katz
Exactly Well, it wouldn't be overengineered if you didn't have to like you know, start over a couple times.

59:55.38
Bogdan Kharchenko
Yes.

01:00:02.64
Chris Morrell
That's fair. That's fair, well I feel like we we literally just hit 1 hour and it kind of feels like this is a good place to start wrapping things up. Um I I am I wanted I want to spend a little time just like thinking about it I'm not. I'm I'm I'm only 90% convinced. Ah and I I want to see if anybody else who who we've kind of talked to about maybe ah coming on for an episode if they listen to this and say. Well, you didn't even talk about you know'd I'd be curious if there's some other strategy that none of us have thought of um and that's kind of the concept right? like I'm thinking of this as a serial podcast of mini so of miniseries right? So each topic is. Sort of going to be its own season and we're hopefully going to have a couple episodes on each topic. Um, you know that's going to be determined by how interesting that is I don't I obviously don't want to have the same conversation over and over with different people. Um, so it's not 100% over but I feel pretty good about where where we landed how about you 2 yeah.

01:01:15.14
Skyler Katz
Yeah I like it I Ah this feels like what I was what I was hoping we'd come to this this agreement. So.

01:01:23.72
Bogdan Kharchenko
Yeah I think ah this format I feel like ah the challenge that we have you know on our daily you know work schedule is you know we don't allocate that much time like giving too much deep thought maybe individually but I feel like. Coming together and thinking about these ideas and looking at some pros and cons as is a great ah solution and I really hope that the topics that we you know the the packages that we looked at the ideas that we had are challenged by other people because you know. I know that we we will only scratch the surface on this I can imagine 50 other mechanisms of doing the same thing. So Awesome. So.

01:02:02.49
Chris Morrell
Ah, well so I have a small list of things already I know for sure. Um, maybe it's not going to be episode or season 2 or however, we we structure it but I know for sure the idea of how do you reference. Data from code that needs to be in the database like I I definitely want to approach that um, because I have I think I have landed on on something that I like but um I bump into this. Another another friend of mine that I I like hop on pairing sessions with like it has come up multiple times on the app that he's working on like and and we just we deal with it all the time. So I definitely want to get there I also feel like um, thinking about. Custom relationships custom eloquent relations is something that ah I am really interested in. Um, but I don't know way way overthinking is there anything else that.

01:03:14.25
Bogdan Kharchenko
Yeah I Just want to add? Yeah I I Just want to like um I suppose think about what you just said about the referencing data from code in the ah roles and permissions world right? after we've done all this seating Somehow. Ah, we want to be able to reference like what is an admin and how can we resolve that to an actual model.. That's what you're referring to as a role. Yeah.

01:03:37.83
Chris Morrell
Well yeah I mean if your if your roles and permissions set up is truly like truly 100% configurable then maybe not right because you if you need.

01:03:53.22
Bogdan Kharchenko
So okay, so.

01:03:55.91
Chris Morrell
You know if you need to be able to set set up any role and any permission so roles and permissions maybe isn't a great example but like an example that that ah just to kind of be thinking about is like imagine that you are I was looking at ah a video on laircast about building like a product. Dock tracker right? like if you wanted to build something that lets you keep track of like who has playstation 5 s in stock across these like 10 vendors right? Um, and and and this is this like ah this was a real series on layer cast where. Jeffrey Way was kind of walking through how you would do that and 1 of the pieces. There is well I want say best buy and Amazon and you know some other and and gamestop or whatever as vendors in the database somewhere. But also going to need to specifically reference best buy in my code because I need like some special best buy Crawler or something that's specific to that vendor. So it's like I know that there is a vendor record in a vendor's table somewhere. Represents best buy and I need to load that up so that I can create new records that are associated with that database model and so like am I just going to hard code like this is the id in the database for best buy.

01:05:29.92
Chris Morrell
Or am I going to just look up. Um the vendor whose name is best buy and hope that that that that vendor's name or you know the way I I wrote the name never changes I I never. Need to go back in and add like a registered trademark to it or you know, um, something like that that I think that's a better example.

01:05:54.42
Bogdan Kharchenko
Right? Okay, yeah, yeah, now that ah that definitely is a challenge I Think for many people. Awesome! okay.

01:06:02.61
Chris Morrell
Yeah, um, are there any other are there any other things that we've talked about that kind of fall in that category that we can just kind of leavea as as stuff that stuff for us to think about or for anyone to kind of um. Ponder as they're an hour 6 minutes into this podcast if anyone ever gets an hour 6 minutes into this podcast.

01:06:26.89
Skyler Katz
I don't I mean I'm just I'm curious what other people what other people think or feel or like what things they wish they could over engineer.

01:06:37.59
Chris Morrell
Yeah, 100% Everyone has had this experience of like God I do I have to do this all the time and I wish there was a better way. But I've never had an excuse to actually figure out. The best way to.

01:07:00.85
Chris Morrell
All right? Well ah with that I guess I'm gonna I'm gonna click this this button and we're gonna stop sound good all right? Thank you guys.

01:07:07.65
Skyler Katz
Sounds great.

01:07:08.30
Bogdan Kharchenko
That sounds good. See you guys next time.