| *** ctbruce has joined #trustable | 08:31 | |
| *** sambishop has joined #trustable | 08:45 | |
| *** laurenceurhegyi has joined #trustable | 09:31 | |
| *** tiagogomes has joined #trustable | 09:49 | |
| * jmacs finds the git-candidate scenarios list, which is a good example of nontrivial BDD: https://bitbucket.org/richardipsum/git-candidate/src/4226e8e4425bbb484ad017b63d7463251813be6e/scenarios?at=master&fileviewer=file-view-default | 11:22 | |
| persia | A nice thing about that is it embeds an example of a requirement that was dropped. | 11:30 |
|---|---|---|
| *** laurenceurhegyi has quit IRC | 13:18 | |
| *** laurenceurhegyi has joined #trustable | 13:19 | |
| *** sambishop has quit IRC | 14:14 | |
| *** sambishop has joined #trustable | 14:29 | |
| *** sambishop has quit IRC | 14:46 | |
| *** sambishop has joined #trustable | 14:46 | |
| *** AlisonChaiken has quit IRC | 15:48 | |
| *** laurenceurhegyi has quit IRC | 16:37 | |
| *** laurenceurhegyi has joined #trustable | 17:01 | |
| *** ctbruce has quit IRC | 17:21 | |
| ChrisPolin | Reflecting on the Trustable framework posed earlier (), the absence of any kind of deployment or CI was noted, and we haven't put a lot of thought into how it might be implemented. I don't have a tremendous amount of experience in this area, so any thoughts to steer my thinking would be greatly appreciated. | 17:25 |
| ChrisPolin | s/()/(https://gitlab.com/trustable/overview/wikis/TrustableSystemStructure) | 17:25 |
| persia | Heh, I have lots of opinions on that sort of thing. I hope some of them may be useful. | 17:26 |
| persia | Do you have a good mental mapping of the differences between CI, CD, patch trackers, automergers, and similar? | 17:26 |
| ChrisPolin | Not especially, no | 17:27 |
| persia | My nomenclature may differ slightly from others, and there is a lot of overlap when discussing specific products, but loosely I use the following nomenclature. | 17:27 |
| persia | A patch tracker is a system by which developers can submit candidate patches for review by other developers. Examples include patchwork, gerrit, gitlab/github "merge requests", email lists to which folk submit patches, etc. | 17:28 |
| persia | Generally speaking, a patch tracker contains some mechanism by which other developers can effectively receive the patch, and provide feedback as to whether the patch is good, if it needs changing, if they think it ought be merged, etc. | 17:29 |
| persia | Some patch trackers have additional functionality, but I consider that outside the scope of patch tracking, as such. | 17:29 |
| persia | A Continuous Integration system enables developers to submit code to some central location, after which some processing (typically testing) is performed. | 17:30 |
| persia | Continuous Integration systems come in two flavours, being "pre-merge" and "post-merge". In a "pre-merge" system, code is integrated, built (assuming a build step), and tested (assuming defined tests) before being considered part of the shared active development branch. | 17:32 |
| persia | In a "post-merge" system, code is integrated, built, and tested after being considered part of the shared active development branch. | 17:32 |
| persia | These systems range from fairly complex (zuul, jenkins) to fairly simple (post-merge git hook on the "master" branch). | 17:33 |
| persia | "build" activities are usually considered core to CI, and the phrase "break the build" is often used if someone manages to merge something that the CI detects as inappropriate. | 17:33 |
| persia | "test" activities are almost always included (although not strictly covered by "integration"), to demonstrate that the software not only builds, but still does what it is supposed to do. | 17:33 |
| persia | Typical tests applied in CI systems are fairly lightweight, to reduce the turnaround time between the developer making a change and that developer receiving feedback on the change. More extensive tests may be required for production deployment. | 17:34 |
| persia | A Continuous Deployment system provides a means to define a set of "pipelines", each of which tracks a given revision of a codebase through several steps. Each step may involve some processing (e.g. a "build" stage or a "test" stage), transport (e.g. a "delivery" stage), or modification stage (e.g. "burn to CD" or "sign for authenticated distribution"). | 17:36 |
| persia | Except in rare cases, CD systems will not progress a codebase on a pipeline unless all previous stages have completed successfully, so that one can, for example, ensure that no revision of the codebase that doesn't compile is ever submitted to the test harness. | 17:37 |
| persia | An automerger is a robot reviewer, usually interacting with some sort of patch tracker. When a given candidate patch meets the rules defined by the automerger, that patch will be applied to the automerger's active development branch (which is presumed to be consumed by most developers) | 17:38 |
| persia | automergers tend to have simple rules, and there are few independent automerging systems (zuulv1 is a rare example, current zuul is not an example). gerrit, zuul, gitlab, jenkins, and most other moderately-robust systems in this space will have an inbuilt automerger. | 17:38 |
| persia | Does that help clarify the vocabulary? | 17:39 |
| ChrisPolin | Very much so, thank you. | 17:39 |
| persia | Are there other systems that you have heard in this context that it would be helpful to describe? | 17:39 |
| ChrisPolin | It's been primarily Patch Trackers and CI that I've been reading into, so that covers it. | 17:40 |
| persia | In the context of TSS, I think some CD is also appropriate. | 17:41 |
| persia | I'm a huge fan of pre-merge CI, because one can never reach a "break the build" state that way. | 17:41 |
| persia | But this means that one doesn't necessarily want to do all the processing for every commit, because there's any number of them that will never merge. | 17:41 |
| ChrisPolin | Yes, I agree. The idea of tracking a revision through the pipeline is essentially what the TSS has been designed for. | 17:42 |
| persia | So the transformation of Requirements into OpenControl, and especially the prodution of SSP might be something that makes more sense with CD than with CI. | 17:42 |
| persia | Careful about the multiple uses of "pipeline" in that context. | 17:42 |
| persia | If you assume one developer that never makes mistakes, it's safe to map a classic development pipeline to a CD pipeline. | 17:43 |
| ChrisPolin | Yes, that distinction makes sense now, I was using it in the CD context. | 17:43 |
| persia | But for the sort of systems where TSS is meaningful, that's a dangerous assumption. | 17:43 |
| persia | Also, in cases where there are expensive tests (long-running, require scarce equipment, involves physical destruction of artifacts, etc.), those are something else that is sensible to have as CD, so that only that which has passed CI, and met some other set of conditions actually becomes part of that test. | 17:44 |
| persia | One contrived example of an expensive test is certification of tilt management for certain automobiles, wherein the active suspension computational system needs to dampen responses in a physcial vehicle (and the certification test requires the physical vehicle, to avoid specifying anything about the implementation). | 17:46 |
| ChrisPolin | Yes, that makes sense. | 17:46 |
| persia | Contrariwise, unless you want to define lots of loop pipelines to handle subintegration of feature branches, you also want to have CI, so that you don't clog the CD pipelines with candidates that cannot be consumed. | 17:48 |
| persia | (note that sometimes you want both to accomplish a specific goal: there's a case in OpenStack wherein the translations CI system involves a loop pipeline to run the candidate translations against the last-merged-to-master branch of the translated software) | 17:49 |
| persia | Err, CD loop pipeline. | 17:49 |
| persia | That odd merged beast is then deployed to a non-production location so that the result can be reviewed without translators needing to be able to deploy/test the software themselves. | 17:50 |
| persia | (assuming the translations apply: this is one of several assumptions that are tested in the CD pipeline) | 17:50 |
| persia | So, at a high level, in my mind, a potential solution to the entire model is as follows: | 17:54 |
| persia | Standards, Requirements, Code all live in repos. | 17:54 |
| persia | Requirements == Tests (assuming you can follow BDD) | 17:55 |
| persia | Standards == Tests (assuming you can mechanically validate compliance) | 17:55 |
| persia | Any change to standards, requirements, or code is done as a patch (or patchset), which is submitted to a patch tracker. | 17:55 |
| persia | The patch tracker is integrated with some CI system that reports what (if anything) is broken by that change. | 17:56 |
| persia | (note that a better patch tracker has a means by which developers can identify cross-repo patch dependencies) | 17:56 |
| persia | The CI system runs the tests, and the results are posted. Some rules are defined for an automerger that involve both human review and passing tests. | 17:57 |
| persia | THere is some CD system that consumes the automerger branch, which produces the SSP, so it is always up-to-date with the currently available integrated code. | 17:58 |
| persia | That same CD system also does things like ensure the automerger branch is mirrored to well-known locations, delivers produced artifacts for expensive tests, etc. | 17:58 |
| persia | Does that seem sensible? Were you thinking of something different? | 17:59 |
| persia | (note that I'm assuming that it is acceptable not to merge a requirements change until the code supports the new requirement and fails to support the old requirement) | 17:59 |
| ChrisPolin | No that's very much the line of thinking. My question to you was going to be how these techniques could most effectively be integrated into the model, but that about covers it. | 18:00 |
| ChrisPolin | One point that was raised however | 18:00 |
| persia | Yes? | 18:00 |
| ChrisPolin | Was that the current model is reliant on the BDD standards/requirements == tests paradigm | 18:01 |
| ChrisPolin | Which is ideal for projects that conform to that format. | 18:01 |
| persia | My basic position here is that if we do not have any means to test the requirements, we cannot know if they are met. | 18:01 |
| ChrisPolin | Agreed. | 18:02 |
| persia | Whether this is expressed in Given/When/Then or some other format isn't very important to me. I happen to live Given/When/Then, because I think it maps well to how many people think, but I accept that some folk find it easier to define requirements in other formats. | 18:02 |
| ChrisPolin | I guess the model isn't particularly sensitive to how requirements are mapped to tests, so long as they are done so reliably. | 18:03 |
| persia | I don't believe that the BDD principle asserts that we must use Given/When/Then, but only that we must test the *behaviour*, rather than the prior TDD model, wherein one tested the *code*. I think "behaviour" and "requirements" map almost perfectly in this context. | 18:03 |
| ChrisPolin | True. | 18:04 |
| * persia once wrote "The problem with TDD is that developers write test to make sure the code does what they think it should do, rather than tests that see if the code meets the requirements.", which is another way to say the same thing. | 18:04 | |
| ChrisPolin | Yes, true. Ok, that sounds like a sensible model in which case. | 18:05 |
| persia | The key thing is to make sure that the people who care about the standards agree with the machine-testable representation of the standard, and that the people who care about the requirements agree with the machine-testable representation of the requirements. | 18:07 |
| persia | (which may require creation of software artifacts that represent textual standards/requirements with some validation mechanism to map that software to the originating documents) | 18:08 |
| persia | Unfortunately, I don't believe that can be done except in a manual manner, as if one starts with something that cannot be machine-processed to derive tests, one probably also has something that cannot be machine-processed to verify correct interpretation of a machine-processable edition. | 18:09 |
| * persia seems to have become confused about levels of parenthesis, and apologises for any confusion | 18:10 | |
| ChrisPolin | I follow. I think there will always be some aspects that can and can't be tested automatically for with standards/requirements. | 18:11 |
| persia | As long as standards and requirements are written in a machine-compatible format, all should be fine. This is part of why I like G/W/T: my prior experience with UML to generate machine-compatible requirements seems to quickly get away from the comprehensible comfort zone of many people. | 18:13 |
| persia | To be fair, folk doing standards have more inventive to dig into things, so UML might be suitable for them. | 18:14 |
| persia | s/inventive/incentive/ | 18:16 |
| ChrisPolin | Mm, although if G/W/T is more widely inclusive while still as effective, it makes sense to adopt it. | 18:17 |
| persia | Yes, but only as a useful tool, rather than an enforced standard. I believe the standard should simply require formatting requirements in such a way that they can be processed by machine to cause testing to occur, which tests are able to confirm if the requirements are met. | 18:19 |
| ChrisPolin | Oh of course. | 18:19 |
| ChrisPolin | This has been hugely useful. | 18:20 |
| persia | Cool. I like to think about these things, so feel free to poke me if you have other thoughts. | 18:20 |
| ChrisPolin | Rest assured, I have plenty of reading to do, and there will be poking in due course. | 18:20 |
| persia | Sadly, I have yet to find a technology stack that meets my expectations in this space, so I'm really looking forward to more folk wanting to do it sensibly, in the hopes that this helps cause such a stack to exist. | 18:21 |
| ChrisPolin | That's very much the initiative of the project. | 18:22 |
| ChrisPolin | s/initiative/incentive | 18:22 |
| persia | Oh! I thought this was about defining a standard against which tooling could be tested. | 18:23 |
| persia | Oops. I read that as "intent" | 18:23 |
| ChrisPolin | It is largely about raising the discussion around the philosophical interpretation and subsequent objective metrics that define 'Trustable Software'. | 18:25 |
| ChrisPolin | So the end goal would ideally be a defined design framework (I'm reluctant to use the word 'standard', due to the connotations in this context), from which software can be created that can be objectively deemed 'Trustable'. | 18:28 |
| ChrisPolin | Whether this involves bespoke software or simply a process using existing software (likely a combination of both) is still emerging. | 18:29 |
| persia | I think that last distinction distracts. | 18:30 |
| ChrisPolin | Which distinction, re: software? | 18:30 |
| persia | My concept of the end goal matches that of a defined design framework. I think that design framework should itself be machine-readable and machine-testable. | 18:30 |
| persia | Yes, re:software | 18:30 |
| ChrisPolin | Ideally yes. | 18:31 |
| persia | With the presence of such a definition, one could objectively determine if a given tool or collection of tooling met the requirements. | 18:31 |
| persia | At that point, if there is no collection of tooling that meets the requirements, it would be interesting to start one, but also interesting to report the failed tests to other tools, so that there could be multiple implementations. | 18:32 |
| *** laurenceurhegyi changes topic to "Trustable software discussion. See https://lists.trustable.io/pipermail/trustable-software/ for more context. Discussions on this channel will be logged publically in https://irclogs.baserock.org/trustable/" | 18:32 | |
| ChrisPolin | Mm, I'm perhaps putting the cart before the horse. | 18:32 |
| persia | This A) reduces the risk that a change to *THE* trusted trustable toolchain inadvertantly causes false positive reports, and B) allows people to take tooling from a variety of vendors, with some means to ensure that all toolchains offered will meet the needs. | 18:33 |
| ChrisPolin | True. | 18:33 |
| ChrisPolin | Initial focus must then be on the model. | 18:34 |
| persia | I hadn't thought of it quite that way, but yes, if you presume there will be THE solution, then yes, cart-before-horse. My thought pattern was more cart-before-road (wherein one doesn't know if it is paved or railed, how wide, etc.) | 18:34 |
| ChrisPolin | Probably a better metaphor, yep. | 18:35 |
| ChrisPolin | I think I have a fair bit of thinking to do here. I'll spend tomorrow rearranging the TSS a little to incorporate this new information, and see if it's any closer to the mark. | 18:37 |
| laurenceurhegyi | https://trello.com/b/i09K3f5X/trustable-phase-1 | 18:43 |
| laurenceurhegyi | paulsherwood ^^ | 18:43 |
| *** tiagogomes has quit IRC | 19:06 | |
| *** laurenceurhegyi has quit IRC | 19:48 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!