IRC logs for #trustable for Monday, 2016-12-19

*** ctbruce has joined #trustable08:31
*** sambishop has joined #trustable08:45
*** laurenceurhegyi has joined #trustable09:31
*** tiagogomes has joined #trustable09:49
* jmacs finds the git-candidate scenarios list, which is a good example of nontrivial BDD:
persiaA nice thing about that is it embeds an example of a requirement that was dropped.11:30
*** laurenceurhegyi has quit IRC13:18
*** laurenceurhegyi has joined #trustable13:19
*** sambishop has quit IRC14:14
*** sambishop has joined #trustable14:29
*** sambishop has quit IRC14:46
*** sambishop has joined #trustable14:46
*** AlisonChaiken has quit IRC15:48
*** laurenceurhegyi has quit IRC16:37
*** laurenceurhegyi has joined #trustable17:01
*** ctbruce has quit IRC17:21
ChrisPolinReflecting on the Trustable framework posed earlier (), the absence of any kind of deployment or CI was noted, and we haven't put a lot of thought into how it might be implemented. I don't have a tremendous amount of experience in this area, so any thoughts to steer my thinking would be greatly appreciated.17:25
persiaHeh, I have lots of opinions on that sort of thing.  I hope some of them may be useful.17:26
persiaDo you have a good mental mapping of the differences between CI, CD, patch trackers, automergers, and similar?17:26
ChrisPolinNot especially, no17:27
persiaMy nomenclature may differ slightly from others, and there is a lot of overlap when discussing specific products, but loosely I use the following nomenclature.17:27
persiaA patch tracker is a system by which developers can submit candidate patches for review by other developers.  Examples include patchwork, gerrit, gitlab/github "merge requests", email lists to which folk submit patches, etc.17:28
persiaGenerally speaking, a patch tracker contains some mechanism by which other developers can effectively receive the patch, and provide feedback as to whether the patch is good, if it needs changing, if they think it ought be merged, etc.17:29
persiaSome patch trackers have additional functionality, but I consider that outside the scope of patch tracking, as such.17:29
persiaA Continuous Integration system enables developers to submit code to some central location, after which some processing (typically testing) is performed.17:30
persiaContinuous Integration systems come in two flavours, being "pre-merge" and "post-merge".  In a "pre-merge" system, code is integrated, built (assuming a build step), and tested (assuming defined tests) before being considered part of the shared active development branch.17:32
persiaIn a "post-merge" system, code is integrated, built, and tested after being considered part of the shared active development branch.17:32
persiaThese systems range from fairly complex (zuul, jenkins) to fairly simple (post-merge git hook on the "master" branch).17:33
persia"build" activities are usually considered core to CI, and the phrase "break the build" is often used if someone manages to merge something that the CI detects as inappropriate.17:33
persia"test" activities are almost always included (although not strictly covered by "integration"), to demonstrate that the software not only builds, but still does what it is supposed to do.17:33
persiaTypical tests applied in CI systems are fairly lightweight, to reduce the turnaround time between the developer making a change and that developer receiving feedback on the change.  More extensive tests may be required for production deployment.17:34
persiaA Continuous Deployment system provides a means to define a set of "pipelines", each of which tracks a given revision of a codebase through several steps.  Each step may involve some processing (e.g. a "build" stage or a "test" stage), transport (e.g. a "delivery" stage), or modification stage (e.g. "burn to CD" or "sign for authenticated distribution").17:36
persiaExcept in rare cases, CD systems will not progress a codebase on a pipeline unless all previous stages have completed successfully, so that one can, for example, ensure that no revision of the codebase that doesn't compile is ever submitted to the test harness.17:37
persiaAn automerger is a robot reviewer, usually interacting with some sort of patch tracker.  When a given candidate patch meets the rules defined by the automerger, that patch will be applied to the automerger's active development branch (which is presumed to be consumed by most developers)17:38
persiaautomergers tend to have simple rules, and there are few independent automerging systems (zuulv1 is a rare example, current zuul is not an example).  gerrit, zuul, gitlab, jenkins, and most other moderately-robust systems in this space will have an inbuilt automerger.17:38
persiaDoes that help clarify the vocabulary?17:39
ChrisPolinVery much so, thank you.17:39
persiaAre there other systems that you have heard in this context that it would be helpful to describe?17:39
ChrisPolinIt's been primarily Patch Trackers and CI that I've been reading into, so that covers it.17:40
persiaIn the context of TSS, I think some CD is also appropriate.17:41
persiaI'm a huge fan of pre-merge CI, because one can never reach a "break the build" state that way.17:41
persiaBut this means that one doesn't necessarily want to do all the processing for every commit, because there's any number of them that will never merge.17:41
ChrisPolinYes, I agree. The idea of tracking a revision through the pipeline is essentially what the TSS has been designed for.17:42
persiaSo the transformation of Requirements into OpenControl, and especially the prodution of SSP might be something that makes more sense with CD than with CI.17:42
persiaCareful about the multiple uses of "pipeline" in that context.17:42
persiaIf you assume one developer that never makes mistakes, it's safe to map a classic development pipeline to a CD pipeline.17:43
ChrisPolinYes, that distinction makes sense now, I was using it in the CD context.17:43
persiaBut for the sort of systems where TSS is meaningful, that's a dangerous assumption.17:43
persiaAlso, in cases where there are expensive tests (long-running, require scarce equipment, involves physical destruction of artifacts, etc.), those are something else that is sensible to have as CD, so that only that which has passed CI, and met some other set of conditions actually becomes part of that test.17:44
persiaOne contrived example of an expensive test is certification of tilt management for certain automobiles, wherein the active suspension computational system needs to dampen responses in a physcial vehicle (and the certification test requires the physical vehicle, to avoid specifying anything about the implementation).17:46
ChrisPolinYes, that makes sense.17:46
persiaContrariwise, unless you want to define lots of loop pipelines to handle subintegration of feature branches, you also want to have CI, so that you don't clog the CD pipelines with candidates that cannot be consumed.17:48
persia(note that sometimes you want both to accomplish a specific goal: there's a case in OpenStack wherein the translations CI system involves a loop pipeline to run the candidate translations against the last-merged-to-master branch of the translated software)17:49
persiaErr, CD loop pipeline.17:49
persiaThat odd merged beast is then deployed to a non-production location so that the result can be reviewed without translators needing to be able to deploy/test the software themselves.17:50
persia(assuming the translations apply: this is one of several assumptions that are tested in the CD pipeline)17:50
persiaSo, at a high level, in my mind, a potential solution to the entire model is as follows:17:54
persiaStandards, Requirements, Code all live in repos.17:54
persiaRequirements == Tests (assuming you can follow BDD)17:55
persiaStandards == Tests (assuming you can mechanically validate compliance)17:55
persiaAny change to standards, requirements, or code is done as a patch (or patchset), which is submitted to a patch tracker.17:55
persiaThe patch tracker is integrated with some CI system that reports what (if anything) is broken by that change.17:56
persia(note that a better patch tracker has a means by which developers can identify cross-repo patch dependencies)17:56
persiaThe CI system runs the tests, and the results are posted.  Some rules are defined for an automerger that involve both human review and passing tests.17:57
persiaTHere is some CD system that consumes the automerger branch, which produces the SSP, so it is always up-to-date with the currently available integrated code.17:58
persiaThat same CD system also does things like ensure the automerger branch is mirrored to well-known locations, delivers produced artifacts for expensive tests, etc.17:58
persiaDoes that seem sensible?  Were you thinking of something different?17:59
persia(note that I'm assuming that it is acceptable not to merge a requirements change until the code supports the new requirement and fails to support the old requirement)17:59
ChrisPolinNo that's very much the line of thinking. My question to you was going to be how these techniques could most effectively be integrated into the model, but that about covers it.18:00
ChrisPolinOne point that was raised however18:00
ChrisPolinWas that the current model is reliant on the BDD standards/requirements == tests paradigm18:01
ChrisPolinWhich is ideal for projects that conform to that format.18:01
persiaMy basic position here is that if we do not have any means to test the requirements, we cannot know if they are met.18:01
persiaWhether this is expressed in Given/When/Then or some other format isn't very important to me.  I happen to live Given/When/Then, because I think it maps well to how many people think, but I accept that some folk find it easier to define requirements in other formats.18:02
ChrisPolinI guess the model isn't particularly sensitive to how requirements are mapped to tests, so long as they are done so reliably.18:03
persiaI don't believe that the BDD principle asserts that we must use Given/When/Then, but only that we must test the *behaviour*, rather than the prior TDD model, wherein one tested the *code*.  I think "behaviour" and "requirements" map almost perfectly in this context.18:03
* persia once wrote "The problem with TDD is that developers write test to make sure the code does what they think it should do, rather than tests that see if the code meets the requirements.", which is another way to say the same thing.18:04
ChrisPolinYes, true. Ok, that sounds like a sensible model in which case.18:05
persiaThe key thing is to make sure that the people who care about the standards agree with the machine-testable representation of the standard, and that the people who care about the requirements agree with the machine-testable representation of the requirements.18:07
persia(which may require creation of software artifacts that represent textual standards/requirements with some validation mechanism to map that software to the originating documents)18:08
persiaUnfortunately, I don't believe that can be done except in a manual manner, as if one starts with something that cannot be machine-processed to derive tests, one probably also has something that cannot be machine-processed to verify correct interpretation of a machine-processable edition.18:09
* persia seems to have become confused about levels of parenthesis, and apologises for any confusion18:10
ChrisPolinI follow. I think there will always be some aspects that can and can't be tested automatically for with standards/requirements.18:11
persiaAs long as standards and requirements are written in a machine-compatible format, all should be fine.  This is part of why I like G/W/T: my prior experience with UML to generate machine-compatible requirements seems to quickly get away from the comprehensible comfort zone of many people.18:13
persiaTo be fair, folk doing standards have more inventive to dig into things, so UML might be suitable for them.18:14
ChrisPolinMm, although if G/W/T is more widely inclusive while still as effective, it makes sense to adopt it.18:17
persiaYes, but only as a useful tool, rather than an enforced standard.  I believe the standard should simply require formatting requirements in such a way that they can be processed by machine to cause testing to occur, which tests are able to confirm if the requirements are met.18:19
ChrisPolinOh of course.18:19
ChrisPolinThis has been hugely useful.18:20
persiaCool.  I like to think about these things, so feel free to poke me if you have other thoughts.18:20
ChrisPolinRest assured, I have plenty of reading to do, and there will be poking in due course.18:20
persiaSadly, I have yet to find a technology stack that meets my expectations in this space, so I'm really looking forward to more folk wanting to do it sensibly, in the hopes that this helps cause such a stack to exist.18:21
ChrisPolinThat's very much the initiative of the project.18:22
persiaOh!  I thought this was about defining a standard against which tooling could be tested.18:23
persiaOops.  I read that as "intent"18:23
ChrisPolinIt is largely about raising the discussion around the philosophical interpretation and subsequent objective metrics that define 'Trustable Software'.18:25
ChrisPolinSo the end goal would ideally be a defined design framework (I'm reluctant to use the word 'standard', due to the connotations in this context), from which software can be created that can be objectively deemed 'Trustable'.18:28
ChrisPolinWhether this involves bespoke software or simply a process using existing software (likely a combination of both) is still emerging.18:29
persiaI think that last distinction distracts.18:30
ChrisPolinWhich distinction, re: software?18:30
persiaMy concept of the end goal matches that of a defined design framework.  I think that design framework should itself be machine-readable and machine-testable.18:30
persiaYes, re:software18:30
ChrisPolinIdeally yes.18:31
persiaWith the presence of such a definition, one could objectively determine if a given tool or collection of tooling met the requirements.18:31
persiaAt that point, if there is no collection of tooling that meets the requirements, it would be interesting to start one, but also interesting to report the failed tests to other tools, so that there could be multiple implementations.18:32
*** laurenceurhegyi changes topic to "Trustable software discussion. See for more context. Discussions on this channel will be logged publically in"18:32
ChrisPolinMm, I'm perhaps putting the cart before the horse.18:32
persiaThis A) reduces the risk that a change to *THE* trusted trustable toolchain inadvertantly causes false positive reports, and B) allows people to take tooling from a variety of vendors, with some means to ensure that all toolchains offered will meet the needs.18:33
ChrisPolinInitial focus must then be on the model.18:34
persiaI hadn't thought of it quite that way, but yes, if you presume there will be THE solution, then yes, cart-before-horse.  My thought pattern was more cart-before-road (wherein one doesn't know if it is paved or railed, how wide, etc.)18:34
ChrisPolinProbably a better metaphor, yep.18:35
ChrisPolinI think I have a fair bit of thinking to do here. I'll spend tomorrow rearranging the TSS a little to incorporate this new information, and see if it's any closer to the mark.18:37
laurenceurhegyipaulsherwood ^^18:43
*** tiagogomes has quit IRC19:06
*** laurenceurhegyi has quit IRC19:48

Generated by 2.14.0 by Marius Gedminas - find it at!