Great PR infos provided.

+1 for that.

Setting up the in-memory db is also what I do every time and still I have to look it up more than I like to admit :-) So providing an easier way to use would be awesome.

I think the integration directory would also be an improvement.

The only thing I would change: I wouldn't take an Eloquent model as an example for a unit test because it can be misleading. Yes, you can use it and it is valid, but as an example I would prefer showing testing something else like a plain PHP class.

Not a bad idea, but it seems like all the "real" apps I've worked on aren't able to use sqlite because of the DB structure or renaming columns in a migration somewhere. That being said, this would seem to be a good framework default. Perhaps adding an additional callout or section to the docs about considerations to make when swapping to a regular mysql testing DB would also be useful?

The only thing I would change: I wouldn't take an Eloquent model as an example for a unit test because it can be misleading. Yes, you can use it and it is valid, but as an example I would prefer showing testing something else like a plain PHP class.

@christophrumpel Heya, thanks! I'm gonna keep this part because it's actually one of the main points I want to make here that you can perfectly unit test Eloquent models. Plus there aren't any other skeleton defaults which can be used in a unit test so I'd rather not have to create a dedicated class for it.

Maybe add foreign key checks too if database connection is available?
I usually do it in TestCase::setUp

https://sqlite.org/pragma.html#pragma_foreign_keys

@hulkur heya. Foreign keys for the sqlite database are enabled by default so there's no need to do separate checks for that: https://github.com/laravel/laravel/blob/master/config/database.php#L43

I'm indifferent on the directory changes, but using an in-memory database gives me a little pause.

I'll just mention 1 issue I've run into before that prevents me from using SQLite. Without an extra extension, SQLite is unable to use 'json' type columns.

Personally I just put any integration test in the "Feature" test directory, which was kind of its original purpose. Do you think by introducing a third distinction we are making things more confusing? What's an integration test vs. a feature test? Feature tests are definitely integration tests. Are some integration tests not feature tests?

Do you think by introducing a third distinction we are making things more confusing?

No not at all. Let me clarify: the difference, in my eyes, lies in the cycle it undertakes. Integration tests are tests which test components. Think: saving data in a database, dispatching jobs and checking reactions (mocking events, etc). Because you're testing one or only a few components they're faster than feature tests.

Feature tests, on the other hand are full cycle tests. You call an endpoint and follow the full request through an app and assert behavior. They touch a lot more parts of the app than just a couple components. Usually it involves lots of components being executed. And thus this makes these tests slower than integration tests.

There's a fourth category as well. End-to-End tests, which mimic the behavior of an actual user using your app. Think Dusk tests, actually making payments on a billing page with the testing api of Stripe, saving files on an actual test S3 bucket. This fourth category might be a little much for the default skeleton (mainly also because Dusk scaffolds this group) but I think these three directories definitely have their place.

All in all I want to say that there's definitely a clear distinction between Unit, Integration and Feature tests in my eyes. But I'm all open to listen to counter arguments of course 🙂

Let me show you another example. In Laravel.io I've separated the test suite between "Feature", "Component" (Integration) & spec (Unit) tests. Have a look at these different tests to see how they differ, I think you'll get a better picture on what I mean: https://github.com/laravelio/portal/tree/master/tests

@driesvints I checked your example and i found Component and Feature only. Also i checked one of compnents tests. It seems same idea of Unit Test.

@ahmad-shawky spec tests are done by PhpSpec and live in the spec directory.

Also i checked one of compnents tests. It seems same idea of Unit Test.

Which one?

I also want to note that there's definitely no fixed rule on separating Integration and Feature tests. It's just that I believe they behave differently enough to warrant the separation 🙂

@ahmad-shawky spec tests are done by PhpSpec and live in the spec directory.

Also i checked one of compnents tests. It seems same idea of Unit Test.

Which one?

https://github.com/laravelio/portal/blob/master/tests/Components/Models/ReplyTest.php

@driesvints I understand your idea more clearly now. I think will be good

Just dropping this here: https://blog.twitter.com/engineering/en_us/topics/insights/2017/the-testing-renaissance.html

Integration test - One of a few definitions is possible (thus we’ll avoid this term)

A test covering many servers and verifying that they work together
A test covering many classes and verifying that they work together

I always liked the fact that Laravel avoids that term due to it's ambiguity. Unit tests can also be ambiguous, but in the context of Laravel it was always "if it's not feature tests, its unit test" regardless of what they mean to you or your project.

https://github.com/laravelio/portal/blob/master/tests/Components/Models/ReplyTest.php

@ahmad-shawky that still makes use of a database under the hood, hence why it's an integration test :)

Unit tests can also be ambiguous, but in the context of Laravel it was always "if it's not feature tests, its unit test"

@deleugpn This is something I've always found confusion myself when working on Laravel projects. Hence the PR and the attempt to make things more clear :)

I understand the sentiment, but just to make sure my post wasn't confusing: having two ambiguous test suite is not better than having only one.

I would rather see the Integration tests in the feature folder. They're often confused and I don't think it's Laravel's job to introduce further confusion by having them at the same level. Laravel defaulting to tests/Feature/Integration would be a good starting point and I could even see a tests/Feature/Acceptance or tests/Feature/Dusk possibly.

imo it might be confusing. We're getting trouble to find the difference in this thread. I think if it is really going to be merged it must be clear the difference about the integration and feature tests

Not fussed about the in-memory db default as we use raw queries in our app (on duplicate key update, for example) and have to test it against mySQL. But can see it being useful for like 80% of scenarios.

Love the new Integrations test idea (assuming I'm understanding it right). The way I'm understanding your differentiation is:

1. A feature test might be testing POST /api/v1/users { data } via an api call
2. An integration test would involve testing the creation of the user from, say, a form in the application that hits the same API as above but it would also test the confirmation output too

Have I understood that right?

I really like the separation of Unit, Integration, and Feature test directories.

However, in my experience using SQLite to test your integration against can in some cases lead to false positives. I have encountered multiple issues where my test suite against SQLite ran perfectly fine only to find that I broke my production server which ran on PostgreSQL. I therefore wouldn't personally use that, but I'd say it's fine as a default. You can always change the phpunit settings.

Perhaps a Testcase class can be introduced for unit tests, which already includes the MockeryPHPUnitIntegration trait.

Agree with @denniskoster and @JackEllis , I'd use MySQL/Postgre for the tests too to be the same as the production database. Tests passing on SQLite then failing on prod gives a false sense of security IMO.

In my experience people use the Feature directory for http tests and Unit for testing eloquent models and actual unit tests.

If the goal is "make people stop putting their model integration tests in the unit directory", a better structure would be:

• Integration
  • Models
  • Http
• Unit

+1 on sqlite by default

My two cents on the sqlite vs your production driver: in my experience you just want to swap the db driver for a fast one to run your tests often. But within your CI environment you can choose your production driver to make sure your tests pass on it before it's deployed to production. Or you can use the production driver on select tests where you know the behavior is different.

I wouldn't want my tests to run slowly locally just because I want to make sure it uses the correct driver. That's what CI is for imo.

100% agree with Default set In-memory DB.
+1 for this.

I agree with others that it should only have two folders. Once you go three it introduces too much analysis paralysis and just adds more confusion. But the rest of the stuff I love 100%!

What if we just renamed the "Feature" directory to "Integration"?

default In-memory DB is undoubtedly agreed by all.
So I think PR can be split into 2 and easily merge the In-memory DB part.
Further discussion can be continued for the Integration Directory.

Another possible option would be to add a flag to the Laravel installer and one can explicitly ask the installer to prepare a SQLite based PHPUnit setup or a MySQL/Postgres one.

Well, renaming "Feature" to "Integration" seems to make no difference. I mean, they are pretty similar imo.

About in-memory db as default I think it's a christmas gift lol

My teams have some confusion over what a "feature" test is, but adding a third folder will add even further confusion IMHO. I wouldn't mind if feature were renamed to integration or just left as feature, as I feel these types of tests could be called integration tests.

In other environments (i.e., Rails) an Integration test is similar to what Laravel calls a feature test. The words require some context anyway.

Another example where multiple definitions exist: a System test is also called End to End or Functional test in some frameworks.

The terms can be a bit vague so I personally like:

• Unit - unit tests
• Integration - test how various parts of the app interact
• End to End - browser testing

You're showing them that this way is easier to begin with testing.

Why is it easier? Again, I don't necessarily disagree, but I'd like some reasons behind the argument to switch to an in-memory SQLite database. I think we can all agree it's faster. Are there other benefits? Do these benefits outweigh the potential discrepancies between testing on your production database? Are greener users going to be served better with speed or with accuracy?

I love the idea of having an in-memory database out of the box, it would make spinning up a new app and demoing some tests that use a database a breeze.

I do agree with @browner12 that developers should use caution in using a different engine for tests vs. real application. I've been burned by this in the past, but in practice not too often. Not sure if the onus needs to be on the Laravel framework to stop users from using different engines in testing though.

Would the Laravel team consider making the default database connection SQLite? The experience out of the box would require creating a development database, but might be a nice default option. I know Rails uses SQLite by default out of the box.

Setting up a new Laravel app that has a working database connection with zero database configuration would be awesome IMO.

it would make spinning up a new app and demoing some tests that use a database a breeze

Who are we making these defaults for, though? Someone who wants to show some demos, or somebody starting a production ready application?

Setting up a new Laravel app that has a working database connection with zero database configuration would be awesome IMO.

Maybe there's some pain point I'm missing here? I use Homestead, and it creates the MySQL databases for me. All I need to switch is the "database" env variable. Is it hard to make databases?

Are the in memory tests really faster? I have swapped MySQL for Sqlite in memory a number of times and always found the difference to be negligable, and sometimes slower.

Maybe "demo" was the wrong word, but I just mean that it's nice to laravel new a new app idea and start creating tests and features for it without any friction. The friction is fairly small though.

My thought was that SQLite is a decent development database that you can easily change to other drivers, but could provide a default that makes the following possible, with zero setup friction:

laravel new project
cd project
php artisan migrate

It's just a thought though, just throwing the idea out there since my experience in Rails using SQLite as the default has been a nice experience, and it's easy to change to another driver.

Personally I think trying to over-categorize things is not really useful/productive and mostly just pedantic procrastination of actually productive work.

The term "integration test" is unhelpfully vague IMO, and in practice I have written very few true "pure" unit tests in any Laravel application I've developed. Instead I personally use "unit test" to mean "testing a method on a class" (regardless of how many real collaborators I am using or whether it talks to the file system, database, whatever), and "feature test" to mean "testing an HTTP endpoint".

The only other level of categorization I've ever found to be helpful is "browser tests", which is stuff using Dusk in the context of Laravel. To me, trying to categorize your tests based on whether they "integrate" with anything vs. are pure or whatever is just not really helpful, and is a point of friction/bike-shedding more than anything else.

Instead I find it much more useful to categorize based on the APIs the tests are using, so all Dusk tests get grouped together, all HTTP tests that use APIs like $this->get(...) and return TestResponse instances get grouped together, and all tests that interact directly with your application classes get grouped together.

This makes categorization simple and obvious, and much less subjective.

I think the current defaults are totally fine but if they were going to change at all I would probably either have 3 folders named Browser, Http, and Unit, or just get rid of the subfolders altogether and let people do whatever they want.

Edit: I wrote this without reading the whole thread and realizing the discussion about new directories is already finished, sorry 😆 The argument that integration tests extend the framework test case while "pure" unit tests just extend the regular PHPUnit test case is a fair argument for what it's worth and falls fairly inline with my "categorize based on the APIs you're using" approach 👍

I find myself doing this in every single project because of the speed and the easy it is to start over. Unless you have a really complex query you are fine to go with SQLite in testing. I have a large app with joins and subqueries and tests are built into SQLite. I swap to MySQL when I find a really complex query failing on SQLite , but until that moment I was able to enjoy the SQLite speed breeze.

I would like to mention, as a lot of people did before me, that this is only the default configuration. I mean, if you don't agree with this and you prefer testing in MySQL, you still need to change the phpunit.xml file, even nowadays you have to. So I don't see anything wrong with this approach. I would recommend adding a good explanation about SQLite vs Others at the beginning of the Tests section in the documentation, but that's all.

Apart from using SQLite or MySQL for testing, I would recommend to put some variables in there just to prevent running the tests into your local development database by default. Since there are no variables it will use the ones in the .env file, wapping out the data you'd have during your normal development period.

I think using different drivers make your tests doubtful. They are supposed to give you confidence that things will not break.

I mean, we would not run our tests with PHP 7.4 if I we are deploying 7.3.
Even though a new version may be a little bit faster, just takes a simple mistake to slip one new feature like an arrow function or a typed property and accidentally you have your tests passing and your deployments failing. And this scenario is just one minor version bump of the same thing.

If both the default connection and the default test connection were to be sqlite, that's a completely different story. I see benefit in having a database solution that works out of the box like we do have a cache driver, sessions driver, queue driver, etc.
I think it will be an easier start for the framework and I would not mind changing the database to my project needs as I already do with all the other drivers.

I feel we shouldn't try to carry over the established distinctions between a feature and an integration test to the framework, unless we revisit how we define what a unit test is within a Laravel app (currently that's anything that's not feature). The framework has done a tremendous job in making testing simpler. If anything could use a new label it should be unit imo.

I like the sqlite in-memory default 💯.

a few years ago i would probably have agreed with default of in memory sqlite, but after being bit by that i would advise people not to be using it for integration tests. it does not behave the same as other databases (data constraints - or lack of). if you are running a test suite you should make the environment as close as possible to how it will be running in production, otherwise there isnt really much point imo - what good is it to know that you have 100% test coverage and all tests passing on an in memory db if it doesnt work on the database you actually want to use, thats just misleading

also adding to this with how widespread docker is spinning up a clean mysql/postgres/mssql etc etc etc db doesnt really take any effort

Not much time, therefore in short:

• In-memory DB default -> yes, a good idea for local testing purposes as a sensible default (even if I would change it to a real database to make the environment as same as possible)
• Adding a third test directory doesn't make sense to me because a feature test is basically an integration test. It checks if different modules/components are working fine together.

If you imagine the test pyramid there are unit tests at a very low level, then integration (feature) tests on top of that and at the very top, there might be a "super feature test" (which I'd call functional test). But the third one is very theoretical.

So summary: Rename Feature to Integration. (at least if you just want something to change, otherwise keep it as it is). There is no advantage IMHO.

Update: Saw the directory discussion is already done. So just ignore my pov above ;)

Merged! Thanks for all your feedback on this one people! 🙂

I just want to say thank you to everyone who contributed to the discussion. It was very useful, educational and genuinely refreshing to see so many people calmly and politely make their point.

@driesvints
What if we want to use Laravel Model Factories in the Unit tests?
laravel/framework#30879

@mitesh1409 Move them to the Feature test directory or extend your test with the Laravel TestCase class.

@driesvints
Thanks for your reply.

So I believe if we want to make use of Model Factory feature in unit tests, then importing
"Tests\TestCase" instead of "PHPUnit\Framework\TestCase" is perfectly fine and is not a bad practice.
The only drawback is that it will run slow since it will boot the framework.

Thanks again :)

@mitesh1409 you're entirely free what you want to do in your own app