| *** toscalix has joined #trustable | 09:02 | |
| *** sambishop has joined #trustable | 09:32 | |
| *** sambishop has quit IRC | 09:45 | |
| *** sambishop has joined #trustable | 09:45 | |
| *** sambishop has quit IRC | 09:47 | |
| *** sambishop has joined #trustable | 09:47 | |
| persia | reiterative: I'm going to need to read that diff a few more times to understand the relations between all the things, but a couple quick notes: | 10:27 |
|---|---|---|
| persia | 1) There is a statement that The *t.software* MUST have *t.intents* and/or *t.constraints*, which doesn't work for me. We need those at a system level, but I expect there to be lots of software that doesn't have any explicit intents or constraints. | 10:28 |
| paulsherwood | +1 | 10:29 |
| paulsherwood | could we break this change down into smaller merge requests? | 10:29 |
| persia | 2) There is a statement that A *t.change* MUST identify its <list of properties>. I consider this rather in terms of maintaining the integrity of evidence about those properties, if we have them. | 10:29 |
| persia | If we don't have those properties, then it may still be possible to have the result be trustable, but if we cannot trust the information represented by those properties, we have a deeper problem. | 10:30 |
| persia | paulsherwood: I'm not sure about that. There's not that much of the current document I think should be preserved. | 10:31 |
| reiterative | paulsherwood: I'm happy to break the change down if that helps, but it is deliberately non-destructive with respect to the existing material | 10:42 |
| paulsherwood | are you intending to deprecate the existing material? | 10:43 |
| persia | Some of the key ideas there may be reusable in terms of determining how to structure metrics to evaluate potential policies, but I think most of the text will end up deprecated. | 10:45 |
| reiterative | paulsherwood: For definitions.md, I think yes, but as persia says, there is some useful material that we can re-use or align with the core concepts in a later change | 10:46 |
| persia | reiterative: having figured out how to look at your total change (rather than a single commit therein), I think I am comfortable with core-evidence.md, excepting that I believe that the roles should only be a mapping from some shorthand to some means of determining identity, and that each rule should specify authorised roles (rather than having the roles/permissions specify validity for each thing in the rules). | 10:47 |
| persia | (or maybe I'm reading that incorrectly, and getting confused by some of the core-concepts.md content). | 10:54 |
| persia | Essentially, I think that it makes sense to identify which roles are valid as part of the rule, but that might just be an implementation detail. | 10:54 |
| reiterative | persia: Note that I have made the tag and merge concepts in the core evidence doc children of t.store. Are you OK with that? | 11:01 |
| persia | I think that's an improvement. In practice, if one moves merges or tags between stores, one is effectively crossing an information horizon, which means one ought do this as a new change. | 11:02 |
| persia | Essentially, there is no way for Alice's store to trust the integrity of a merge imported from Bob's store *except* to consider it as a change (or series of changes) submitted by Bob (presumably with additional attribution information that indicates from whom Bob received the changes). | 11:03 |
| reiterative | Having discussed with persia, I am going to remove the example-workflow diagram / text, as it needs more thought / context and isn't needed for the new core-* material | 11:17 |
| *** traveltissues has joined #trustable | 11:19 | |
| persia | I think some of that is useful, but probably a separate change. | 11:22 |
| persia | Or as a one-of-many models, because ost orgs have unique policies and processes. | 11:23 |
| reiterative | Yes, I think it would be better to rework it as a pattern | 11:24 |
| persia | That makes a lot of sense. | 11:24 |
| *** toscalix has quit IRC | 11:43 | |
| persia | Right. I've now had a chance to have a proper think about core-concepts.md. I think it begins to go astray near the beginning, in that I don't think the content of core-evidence,.md is necessarily required, nor does it represent the minimal evidence, but rather that it identifies the set of evidence for which any process must be able to assure integrity in order for the trustability to be evaluated. | 12:34 |
| persia | The key distinction being that the absence of evidence is acceptable, so long as the integrity of that absence is preserved. It may be that without some components of that evidence, it becomes very difficult to make statements about the provenance or evaluate the internal consistency of a history, but that may not be important, depending on the entirety of policy and process used in the development of any given system. | 12:37 |
| persia | Conversely, if any of the evidence identified in core-evidence.md is present without a guarantee of integrity, then we absolutely cannot derive any opinion about whether the output of a process might be trustable. | 12:38 |
| persia | In Summary: I believe I agree with paulsherwood that we can usefully distinguish t.software from t.system, wherein t.software is a collective noun for that subset of artifacts captured by a process that are subject to change, and t.system is the expected result of deploying t.software into some t.environment. | 12:41 |
| persia | For t.store, I'm not sure why it is worth mentioning the intents, constraints, and policies, but if the goal is only to avoid the potential future question "where does one store those things", perhaps it is worth rephrasing as "Where are *t.artifacts*, including those encoding *t.intents*, *t.constraints* and *t.policies* stored". | 12:45 |
| persia | Also for t.store, I don't think how ones process happens to manage tracking of incoming t.changes is necessarily t.store. As an example, the linux kernel tends to be developed by folk sending t.changes to the mailing list, and others sending t.votes to that same mailing list, after which someone collates them into a t.merge and puts them in a t.store (unless t.store is imagined to encapsulate both the SCM repo *AND* the mailing list in this context). | 12:47 |
| persia | I think of t.rules as being about both CI and CD (as not all CI is pre-merge, and not all post-merge CI necessitates automated deployment). | 12:48 |
| persia | On the other hand, maybe my vocabulary is outdated? | 12:48 |
| persia | On t.software requirements: I think the trustability can only be evaluated by looking at both the process and the evidence, rather than evidence alone. | 12:49 |
| persia | I don't think software necessarily has intents or constraints, although systems do (which is why the distinction is useful) | 12:50 |
| persia | I also think it is acceptable for a system to exist without evidence that all contraints are satisfied: it is likely to be a common situation that a system under development does not (yet) satisfy all contraints, and even more likely that a deployed system only satisfies a set of constraints known at deployment time (which is unlikely to match the current constraints if the system is actually being used). | 12:51 |
| * persia is happy with the t.intent requirements | 12:52 | |
| * persia is happy with the t.constraint requirements | 12:52 | |
| persia | the t.environment requirements look reasonable, except that it uses both "t.artefacts", which isn't defined (only t.artifact is defined). As I said yesterday, I now believe that for any reasonable system, the artifacts are only an artefact of the stored history of merges, but we should be consistent in a minimal grammar. | 12:54 |
| persia | For t.change, I feel that repeating the set of expected properties in both this document and core-evidence.md is likely to be a maintenance nightmare: the same applies for several other concepts defined later in the document. | 13:00 |
| persia | I don't think it makes sense to include the bit about complex workflows. Two common patterns in use are the "meta-repo", wherein one store contains definitions that reference specific merges in other stores, used as a means to transactionalise the effects of a collection of merges that happen in multiple repos; the other is the use of "Depends-On" or similar additional information attached to each change, that describes a dependency tree between | 13:03 |
| persia | changes to control how they land. I don't think we want to proscibe either of these (or, for that matter, not allow others, if those technologies exist). | 13:03 |
| persia | I don't understand 'The actions taken in processing a *t.change* SHOULD generate the *t.evidence* required by the applicable *t.policies*'. What is "processing" a change? Is that creating a vote? Creating a merge? Something else? | 13:04 |
| persia | I disagree with 'A *t.change* SHOULD include a text component that summarises its scope, identifies its goals and lists any relevant *t.intents* or *t.constraints*': to me, that is a policy decision. It may be that those are entirely captured in a meta-repo or similar, rather than in individiaul changes to software. | 13:05 |
| persia | I also disagree with the characterisation of a "well-defined" change. To me, those read as policy. | 13:05 |
| reiterative | persia: By 'processing a change' I really meant all the actions that would be covered by a policy, so it's probably redundant | 13:06 |
| persia | I think the t.store requirements should be rephrased to include reference to t.merge in some way (although with t.merge changing to t.store.merge, maybe that gets more complicated). | 13:06 |
| persia | reiterative: Oh, that makes sense. Then yes, I think that should be encapsulated by the requirements for policy, rather than being about t.change | 13:07 |
| persia | t.policy intro text needs to be rewritten without the example. | 13:07 |
| persia | (and some of the first paragraph might better move to t.process anyway) | 13:08 |
| persia | I'm not sure what to do with the example rules. I think that enforces implementation details (although they are only examples). | 13:08 |
| persia | Ideally this should express the requirements about what the rules specify (e.g. every rule needs to be about votes and roles), without necessarily formalising the representation. | 13:09 |
| *** coldtom has quit IRC | 13:09 | |
| persia | For the goal of being able to compare and contrast processes, we probably want the rules in a specific representation, but I think that it ought be possible to construct a competing comparison tool for a set of processes with a different rules representation and still call it "trustable". | 13:10 |
| persia | t.policy also includes some of the details about t.vote: perhaps the document should reference t.vote before t.rule, so that the information can be assumed to already be known | 13:11 |
| persia | (as a general principle, it may make sense to reorder things so that any term that references another term appears after the term referenced: if we have some circular references, it would be good to know (and maybe worth untangling, if possible). | 13:11 |
| persia | I'm also unsure about 'The set of applicable *t.policies* MUST describe a policy that applies to **all** of the participating *t.stores*'. My naïve defintion of "participating store" is a t.store referenced by a t.policy, making this tautalogical. | 13:13 |
| *** coldtom has joined #trustable | 13:13 | |
| reiterative | persia: I was trying to cater for a situation where there were a number of overlapping policies, where at least one of them must control changes to each of the stores that are covered by the full set of policies | 13:15 |
| persia | I believe t.evidence is created when actions are performed on t.change, rather than when a t.change affects a t.artifact. While a t.change generally does affect a t.artifact, much of the evidence is unrelated to any specific artifact, at least as I think of it (where evidence is likely to be properties of the t.change itself, properties of any t.votes associated with the t.change, properties of any t.merge associated with the t.change, and/or any t. | 13:16 |
| persia | artifacts, t.changes, t.identities, etc. referenced by any of those properties). | 13:16 |
| persia | Hrm. I'm not sure I understand "overlapping policies". | 13:17 |
| reiterative | I think I am overcomplicating things | 13:17 |
| reiterative | Thinking about implementation instead of concept | 13:17 |
| persia | In my imagination, any merge actor must only follow a single policy. That policy might be stored in several artifacts. | 13:17 |
| reiterative | Yes, exactly. If we just have one policy, it becomes simpler | 13:18 |
| persia | So if the merge actor is authorised to merge to the store, that implies the policy for that store is the policy used by that merge actor. | 13:18 |
| persia | If a process allows different merge actors with different policies to merge to a single store, I submit that is unlikely to be a trustable policy. | 13:18 |
| persia | Err, trustable process. | 13:18 |
| reiterative | What I really wanted to require was that every store that participates in the policy should be subject to change control | 13:19 |
| persia | For clarity, although I try to push my rhetoric, I'm more interested in ensuring the discussion has been had than in being right (and in some cases will pursue argument where I think I'm wrong, just to make sure it's been thought through), so don't just take my word for these things :) | 13:20 |
| persia | I submit that whether a store is subject to change control is a policy decision. I submit along with that that any store that cannot be represented as a history of merges of changes is unlikely to be trustable. | 13:21 |
| reiterative | That's what I thought you were going to say :-) | 13:21 |
| persia | heh | 13:22 |
| * persia goes back to the detailed critique of core-concepts.md | 13:22 | |
| persia | Does it make sense to require that the integrity of any t.identify information must be able to be assessed, or is that duplicative to the t.evidence requirement? | 13:23 |
| persia | I consider a t.vote to be an assertion of <value> about <label> by <voter>, which may not actually assert evidence is valid and/or present (as it might assert that necessary evidence is invalid or not present). | 13:24 |
| persia | I'm not quite sure how to express that though :( | 13:24 |
| reiterative | I'll have a go | 13:24 |
| persia | Perhaps 'a t.vote is an assertion by a t.identity about a t.change that may be evaluated by t.policy', or is that too terse? | 13:25 |
| reiterative | That's OK, but I'd like to reference t.evidence in there | 13:26 |
| persia | Why? | 13:26 |
| persia | What t.evidence is necessarily relevant to a t.vote? | 13:26 |
| reiterative | If t.vote | 13:28 |
| persia | Consider the case of a roll-call vote for a change in policy by a governing board. The board members are likely to vote based on the content, rather than evidence. Similarly, the necessary evidence for the rule is likely to be whether there was both quorum and a decision (and otherwise be about the board voting model). | 13:28 |
| reiterative | So we could record the outcome of the vote, without the details | 13:28 |
| persia | Well, the submission of votes is likely to generate/contain evidence, so there would be some details (like who voted which way, etc.). | 13:29 |
| persia | It isn't a common model, but there do exist systems by which a secret ballot is expected, which means we specifically don't want the details. | 13:29 |
| reiterative | Yes, but the details might not be considered evidence, but would still inform the vote | 13:29 |
| persia | (although I presume the implementation of that is that there is a single voter that reports a secret ballot was held and the result, where the ballot itself happened outside the information horizon of the t.store). | 13:30 |
| persia | Right. There is very likely to be information that informs the vote that is not normally considered evidence. To some degree, one could claim that the full content of any change was included in "additional_info", but unless there is a means to parse the specific information that was relevant to the voter, it is hard to pin down any specific bit of it. | 13:31 |
| reiterative | But in our model, the t.vote isn't necessarily the kind of vote you are describing. My reasoning for saying that t.vote should be an assertion regarding t.evidence is that you would want the ability to go back and check that evidence. | 13:32 |
| persia | (and similarly hard to indicate that the vote was based on evidence: consider the case where every change must be approved by a project manager as a means of ensuring that the PM updates some external documentation: the PM isn't considering any prior evidence when making votes, but rather making sure the other system is up-to-date and then reporting that back to the store). | 13:32 |
| persia | I think there are two kinds of t.vote. | 13:33 |
| reiterative | Yep. | 13:33 |
| persia | There is the class of t.vote where one can determine the value as a result of mechanical algorithms applied to extant evidence. | 13:33 |
| reiterative | One which relates to evidence, and one which stands as evidnece for data that isn't available as evidence. | 13:33 |
| persia | There is also the class of t.vote where one the process to determine the value cannot (currently) be represented by computation, or for which there is no assertion that the vote is based on evidence. | 13:34 |
| persia | Right. | 13:34 |
| persia | So perhaps "deductive vote" and "opinion vote", athough I usually use the words "robot" and "human" as shorthand. | 13:35 |
| persia | (not that a "robot" vote might not be performed by a human (and many humans like these, especially as a means to game metrics), nor that one cannot construct sufficiently intelligent automation to perform a "human" vote, e.g. by statistical analysis rather than rules-based approaches). | 13:36 |
| reiterative | Objective and subjective? | 13:36 |
| persia | Sure. Those work as well. | 13:36 |
| persia | But I think that whether any particular vote happens to be objective or subjective, and what expectations exist about the entities submitting those votes are entirely policy. I don't think the votes are meaningfully different in terms of date model. | 13:37 |
| reiterative | Agreed | 13:38 |
| persia | For that matter, any logic that is used to produce an objective vote is probably t.software itself, and so ought be tracked appropriately, etc. | 13:38 |
| persia | So, if the language is expected to support policies that permit subjective votes, do you still feel that t.evidence needs to be included in the definition of t.vote? | 13:39 |
| reiterative | Yes for the deductive / objective version, no for the subjective / opinion version | 13:40 |
| reiterative | But actually, the deductive version doesn;t have to relate to evidence | 13:40 |
| reiterative | It might be more useful if it did, but it's not an absolute requirement | 13:41 |
| persia | Depends on how we define evidence. | 13:42 |
| reiterative | So I have: "A t.vote may be objective (representing the outcome of an analytical process performed on a set of data relating to a t.change by the asserting t.identity) or subjective (representing the t.identity's opinion regarding an aspect of the t.change)" | 13:42 |
| persia | If we use the core-evidence.md definition, which includes "additional information" for every object, and we assert that any informaiton not otherwise specified is "additional", then any information that exists in a t.store (whether explicit or implicit) becomes evidence. | 13:43 |
| persia | Do we care about the distinction at this level? | 13:43 |
| persia | We very much do when discussing process. We might when discussing policy. I believe we don't when discussing underlying structure. | 13:43 |
| reiterative | I think evidence is something that we can provide later. A t.vote might relate to data about a change that is not retained, and hence cannot be considered evidence | 13:44 |
| persia | I agree with that. | 13:44 |
| persia | I further assert that evidence is something that we can provide later along with an assertion of a level of confidence about the integrity of that evidence. | 13:44 |
| reiterative | Yes | 13:45 |
| persia | So then we have four kinds of votes. We have ones that are algorithmically determined from evidence, ones that are algorithmically determined by unknown/unrecoverable/untrusted/external/intermediate data, ones with an unknown processing logic based on evidence, and ones with an unknown processing logic based on unknown/unrecoverable/untrusted/external/intermediate data. | 13:47 |
| persia | I expect we can break that down into even more kinds of votes if we try, but I still assert that all types of votes share the same data model and the same requirements, and are consumed by policy in the same manner. | 13:47 |
| * persia takes silence as assent, and presses on | 13:58 | |
| persia | Special roles is tricky, because policy might want to say things like "a vote of 'Approved' may not have the same voter as voted 'Reviewed' or be from the submitter". | 13:59 |
| persia | I'm not sure how to handle that, to some extent because it is less possible to use the trick of using a vote to determine if the votes are correct than it is to use a vote to determine if other evidence is correct. | 14:00 |
| persia | I think the current note is useful, but expect it might be useful to reach out to a wide audience about possible ways to represent special roles to handle various policy choices. | 14:01 |
| persia | I think the detail paragaphs for t.process are good, but I have trouble considering the word "mechanism" in the base definition. | 14:02 |
| persia | Perhaps "means" or "model"? | 14:03 |
| reiterative | "means" works for me | 14:03 |
| persia | It also repeats the implication that t.votes are based on t.evidence. I think it correct to assert that process defines how evidence is produced and assessed, and that it defines how votes are produced and assessed (where the assessment is likely to be the policy), but not that the votes are necessarily a result of assessment of evidence. | 14:04 |
| persia | The example is an excellent way to show the nature of the classes of problems that exist at the intersection of policy and process, and why both concepts are required. | 14:05 |
| persia | Afterthought: it might make sense to reword the three goals in the beginning, to avoid the use of words defined later, where possible (e.g. "software engineering" probably needs to remain, despite the overload of "software|t.software"). The item that most comes to attention is "evidence", for which "properties" might be sufficient substitute. | 15:34 |
| persia | "process" is harder: maybe that needs to stay. "software" when used outside of "software engineering" might want to be "software or systems" or similar. | 15:34 |
| reiterative | I've pushed my updates following persia's detailed review to pa-nomenclature in the repo | 15:37 |
| persia | I think I'm looking at the new version, but have some of the same comments. | 16:05 |
| persia | core-concepts.md:45 seems better left to policy (it's best practice, but I don't think it is required) | 16:06 |
| persia | core-concepts.md:56 appears as part of a long list of potential things that could happen, most of which are not documented in this document | 16:07 |
| persia | Same for core-concepts.md:69, although without that, there are larger issues in validating the integrity of the evidence in general | 16:08 |
| persia | I'm still not delighted by the categorisation of rules in the definition of t.rules, but also still don't know how to reword it to be flexible about implementations | 16:12 |
| persia | Err, that's in t.policy around core-concepts.md:87 | 16:12 |
| persia | And I'd like to revisit the debate about the utility of distinguishing vote types for the text around core-concepts.md:116 | 16:14 |
| persia | I wonder if it is worth adding another paragraph to the top of core-evidence.md that makes it clear that there may be lots of other evidence used for policy evaluation, being either information listed as "additional info" for one of these core elements, or additional elements that may be introduced by a given process. | 16:16 |
| persia | I can imagine that a very large number of processes are going to want to validate that changes address issues or enforce specific constraints, etc., which capability isn't immediately obvious with the current wording. | 16:16 |
| persia | Also in core-evidence.md, t.store.merge individual entries are marked t.merge.*, and similarly for t.store.tag | 16:17 |
| persia | Lastly, I think the patterns change belongs separately, and it is worth describing both the BuildStream and Zuul approaches to cross-repository change dependencies (and more, if we know of them). | 16:18 |
| *** brlogger has joined #trustable | 17:12 | |
| *** brlogger has joined #trustable | 17:17 | |
| *** sambishop has quit IRC | 17:31 | |
| *** traveltissues has quit IRC | 20:03 | |
| *** traveltissues has joined #trustable | 20:33 | |
| *** traveltissues has quit IRC | 20:52 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!