Uses the github.com/ghodss/yaml library to convert YAML to JSON before
unmarshaling for the purposes of supporting YAML and JSON input while
converting to only support JSON internally.
Bug: T207694
Change-Id: I00668014907e9ea54917f5d5067cac08d0668053
Support for mounting a shared volume instead of copying application
files was meant to provide an option for development use cases. This
functionality has never been thoroughly tested or utilized for any use
case. It should be removed for now.
Relates tangentially to task T204591 that simplifies node support.
Change-Id: Ib70cb7bceb504841897a38d732880ba376fe67c8
The builder configuration has proven useful for supporting generic
pre-entrypoint commands such as dependency managers not otherwise
supported by specific Blubber configuration. Adding additional
`builder.requirements` config expands support for such commands by
allowing the user to specify files that should be copied into the image
before the builder command runs.
To support this extra configuration, `builder` had to be changed from a
simple string to a mapping. The builder command must now by given as
`builder.command`.
The pattern of creating parent directories, copying files, and executing
one or more commands prior to the entrypoint has become a common
one. Some of the implementation of this pattern was moved from
`PythonConfig` into shared build macros `build.SortFilesByDir` and
`build.SyncFiles`. All config types that must have requirements files
copied over independently of the entire source tree (`PythonConfig`,
`BuilderConfig`, `NodeConfig`) now delegate to these functions.
Change-Id: I67f33034f22cee2851ec866cfb07ab20c23eba8c
All import paths and other references to the previously Blubber repo in
Phabricator have been changed to point to the new Gerrit project URL.
Note that building or testing against this change will require you to
move your working directory to `$GOPATH/src/gerrit.wikimedia.org/r/blubber`.
Contribution documentation was updated to reflect the changes, and the
`.arcvendor` submodule has been removed.
Bug: T200452
Change-Id: I3ab23e420b2825e86e2bd7c9b3da9d4de23edaa1
Summary:
Implements a rough interface for validating configuration against
arbitrary policy rules. Policies are provided as YAML and passed via the
command line as file paths or remote URIs.
The format of policies is:
enforcements:
- path: <path>
rule: <rule>
Where `<path>` is a YAML-ish path to a config field and `<rule>` is any
expression our config validator understands (expressions built in by the
validator library and custom tags defined in `config.validation.go`).
Example policy:
enforcements:
- path: variants.production.base
rule: oneof=debian:jessie debian:stretch
- path: variants.production.runs.as
rule: ne=foo
- path: variants.production.node.dependencies
rule: isfalse
Command flag parsing was implemented in `main.go` to support the new
`--policy=uri` flag and improve existing handling of `--version` and the
usage statement.
Test Plan: Run `go test ./...`.
Reviewers: thcipriani, demon, hashar, mmodell, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D999
Summary:
A new root and variant `python` config field is provided with two new
fields below, `version` and `requirements`.
The former, `version`, should specify the Python executable to use when
executing related package installation commands and ostensibly the same
executable that will be used to run the application.
The latter, `requirements`, should specify all pip requirements files
such that a compiler that supports layered filesystems (e.g. Docker) can
output separate instructions that will invalidate cache layers for
changes to those files independently of changes to the rest of the
codebase.
Python related instructions will be generated only if either `version`
or `requirements` are given.
Fixes T186545
Test Plan: Run `go test ./...`.
Reviewers: thcipriani, hashar, demon, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Maniphest Tasks: T186545
Differential Revision: https://phabricator.wikimedia.org/D976
Summary:
Introduces new `lives` configuration that provides the name/UID/GID of
the user that will own application files and installed dependencies.
This new configuration is distinct from `runs` in that the former
determines application file location ownership and the latter now only
determines runtime process ownership. Default configuration has also
been introduced for both config sections.
In addition to the new configuration, a new `build.CopyAs` instruction
has been introduced that ensures correct UID/GID ownership of files
copied into the container image, and all unqualified `build.Copy`
instructions are wrapped by the new `build.CopyAs` instruction using the
UID/GID appropriate for the current build phase. A new `build.User`
instruction is also introduced and injected into the build at the start
of certain phases to enforce ownership of `build.Run` processes.
This effective process/file ownership model is:
PhasePrivileged - "root"
PhasePrivilegedDropped - lives.as
PhasePreInstall - lives.as
PhaseInstall - lives.as
PhasePostInstall - runs.as
Fixes T187372
Test Plan: Run `go test ./...`.
Reviewers: thcipriani, hashar, demon, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Subscribers: mmodell
Tags: #release-engineering-team
Maniphest Tasks: T187372
Differential Revision: https://phabricator.wikimedia.org/D984
Summary:
Implemented a validation system using the
`github.com/go-playground/validator` package, extending it with custom
validation tags, and implemented translation of validation errors into
somewhat human-friendly messages.
Fixes T175186
Depends on D845
Test Plan: Run the unit tests and try running blubber against some bad config.
Reviewers: thcipriani, hashar, Jrbranaa, Joe, #release-engineering-team, mobrovac
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Maniphest Tasks: T175186
Differential Revision: https://phabricator.wikimedia.org/D868
Summary:
Fixed all linter warnings and advice except for vet's rule about unkeyed
composite literals which was disabled via a `-composites=false` flag in
`.arclint`. Most unkeyed literals (e.g. `build.Run{"command"}`) in this
project just seem too usefully succinct compared to their more verbose
keyed counterparts.
Depends on D841
Test Plan: Run `arc lint --everything` and verify there are no warnings or advice.
Reviewers: thcipriani, hashar, #release-engineering-team
Reviewed By: thcipriani, hashar, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D845
Summary:
Wrote inline documentation for all the things.
Fixes T168000
Test Plan: Execute `godoc -http=:6060` and proofread all the things.
Reviewers: thcipriani, hashar, #release-engineering-team, demon
Reviewed By: thcipriani, #release-engineering-team, demon
Tags: #release-engineering-team
Maniphest Tasks: T168000
Differential Revision: https://phabricator.wikimedia.org/D841
Summary:
Now that our Phabricator instance handles `go get` requests, we can use
a path without the `.git` suffix as the canonical import path.
Test Plan: Run `go test ./...` and `go build`.
Reviewers: thcipriani, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D780
Summary:
Define `NODE_ENV` based on the configuration and define `NODE_PATH` when
either `npm.install` or `npm.env` are true. The latter behavior is
necessary for final stages of multi-stage builds where npm modules are
not installed but the application requires node-related environment
variables to run.
Depends on D757
Ref T174626
Rename npm module/config to node
Renamed `npm` yaml entry to `node` and `NpmConfig` to `NodeConfig`.
Since we're dealing with general node builds and configuration, this
seems to make more sense.
Test Plan:
Run `go test ./...` or `arc unit`. Run blubber against at least the
`blubber.example.yaml`.
Reviewers: thcipriani, mobrovac, mmodell, #release-engineering-team
Reviewed By: thcipriani, #release-engineering-team
Tags: #release-engineering-team
Maniphest Tasks: T174626
Differential Revision: https://phabricator.wikimedia.org/D759
Summary:
One of Golang's "advantages" is not quibbling over style. To this end
the `go fmt` command exists. This is the result of me running:
find . -name '*.go' -exec go fmt {} \;
Test Plan: built package, ran tests
Reviewers: dduvall, mmodell, #release-engineering-team
Reviewed By: dduvall, mmodell, #release-engineering-team
Subscribers: hashar
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D694
Summary:
Refactored bool config fields to use a new `config.Flag` type that keeps
track of whether it was set by unmarshalled data or merged from another
instance, fixing the behavior of these fields when overwritten by
variants.
Fixes T166353
Test Plan: Run config unit tests (`cd config; go test`)
Reviewers: thcipriani, Joe, hashar, mobrovac, mmodell, #release-engineering-team
Reviewed By: mobrovac
Tags: #release-engineering-team
Maniphest Tasks: T166353
Differential Revision: https://phabricator.wikimedia.org/D680
Summary:
Now that the project is hosted on Phabricator the import URIs must be
changed.
Reviewers: thcipriani, hashar, Jrbranaa, mmodell, #release-engineering-team, demon
Reviewed By: #release-engineering-team, demon
Tags: #release-engineering-team
Differential Revision: https://phabricator.wikimedia.org/D657
Establish phases within Docker compiler to allow providers (apt, npm,
etc.) to inject their own run/copy instructions into to the Dockerfile
compilation process while leaving the compiler agnostic to the providers
themselves.
The instructions and phases are also generalized to leave room for
alternative compilers should they be needed in the future (e.g. aci
support via acbuild) but also as a general design constraint to leave
compiler implementation concerns out of providers.
Reorganized config type declarations.
Added `RunConfig` for declaring working directory and unprivileged
runtime user. A system for enforcing restrictions should probably be
implemented for this.
Implemented Dockerfile compiler, supporting multi-stage output in cases
whether artifacts from other variants are configured.
Tyler Cipriani
Dan Duvall
Gilles Dubuc