# Contributing to Besom

Thank you for contributing to help make Besom better. We appreciate the help!

## Code of Conduct

Please make sure to read and observe our
[Contributor Code of Conduct](CODE-OF-CONDUCT.md).

## Communications

We discuss features and file bugs on GitHub via
[Issues](https://github.com/VirtusLab/besom/issues).

### Issues

Feel free to pick up any existing issue that looks interesting to you
or fix a bug you stumble across while using Besom.
No matter the size, we welcome all improvements.

Before investing a lot of time, please let us know, so we can discuss the issue together.

### Feature Work

For larger features, we'd appreciate it if you open a
[new issue](https://github.com/VirtusLab/besom/issues/new)
before investing a lot of time, so we can discuss the feature together.

Please also be sure to browse
[current issues](https://github.com/VirtusLab/besom/issues)
to make sure your issue is unique, to lighten the triage burden on our maintainers.

### Adding examples

Start with a template that is closest to your use case, and modify it, e.g.:

```bash
mkdir examples/my-example
cd examples/my-example
pulumi --logtostderr new https://github.com/VirtusLab/besom/tree/main/templates/gcp
```

## Branching and versioning strategy

We mostly follow the Pulumi strategy:

- `main` branch contains current `*-SNAPSHOT` version
- `vX.Y.Z` tag marks the `X.Y.Z` release
- `release/vX.Y.Z` branch contains the `X.Y.Z` release
- PRs must have a prefix with the **name of the author and issue number** e.g. `pprazak/123-fix-bug`

> [!NOTE]
> Please make sure to **tag first** before creating a release branch.

Versioning is done using [Semantic Versioning](https://semver.org/), with following additions:

- `x.y.z` for core version, where:
    - `x` no guarantees are made about compatibility,
    - `y` should not break source compatibility,
    - `z` should not break binary compatibility
- `a.b.c-core.x.y` for provider version, where `a.b.c` is the schema version
- `*-SNAPSHOT` versions are used for development versions

## Developing

### Setting up your development environment

You will want to install the following on your machine:

- [Pulumi CLI](https://www.pulumi.com/docs/install/) 3.30.0 or higher
- [Scala-CLI](https://scala-cli.virtuslab.org/install/) 1.0.4 or higher
- JDK 11 or higher (GraalVM version is recommended)
- Scala 3.3.1 or higher
- Go 1.20 or higher
- [protoc](https://grpc.io/docs/protoc-installation/) 24.3 or higher
- [just](https://github.com/casey/just#installation) 1.14.0 or higher
- git 2.37.1 or higher
- unzip
- coursier
- gh

#### Mac OS

```bash
brew install pulumi/tap/pulumi
brew install Virtuslab/scala-cli/scala-cli
brew install coursier/formulas/coursier
brew install just
brew install java11
brew install sbt
brew install go
brew install git
brew install unzip
brew install protobuf # protoc
brew install gh
```

### Preparing a pull request

1. Ensure running `just` passes with no issues.
2. Ensure the branch name is prefixed with your name and contains issue number, e.g. `johndoe/123-fix-bug`.

### Understanding the repo structure

- [`core`](core) contains the core of Scala SDK for Pulumi
- [`besom-cats`](besom-cats) contains the cats effect integration for Scala SDK for Pulumi
- [`besom-zio`](besom-zio) contains the ZIO integration for Scala SDK for Pulumi
- [`codegen`](codegen) contains Scala code generation fo Besom (e.g. used for provider SDKs)
- [`proto`](proto) contains a copy of Pulumi protobuf definitions (generated by a script)
- [`compiler-plugin`](compiler-plugin) contains the compiler plugin for Scala SDK for Pulumi
- [`language-plugin`](language-plugin) contains the Golang language host provider plugin for Scala SDK for Pulumi
- [`templates`](templates) define starter project templates
- [`examples`](examples) contains examples of using Besom
- [`scripts`](scripts) contains various scripts used for development

### Working with local dependencies

There are Just targets that help rebuild all Scala and Go packages from
source. For correct cross-referencing in examples and tests, Scala
packages need to be installed into the local `~/.ivy` repo.

The targets do not yet understand dependencies accurately, so you may
need to re-run to make sure changes to Besom SDK or providers are
rebuilt.

As for Go changes, Pulumi CLI will respect the version of
`pulumi-language-scala` it finds in `PATH` over the default version it
ships with. When testing changes to the language provider, you
therefore might need to manipulate `PATH` to prefer the local version.

#### Publish locally core SDK packages

Publish locally and install necessary Besom packages:

```bash
just publish-local-all 
```

#### Publish locally additional SDKs

You have to generate an SDK for a provider of your choice, use `just cli`, e.g.:

```bash
export GITHUB_TOKEN=$(gh auth token)
just cli packages local azure docker gcp kubernetes random tls
```

To generate all Provider SDKs (takes a very long time):

```bash
export GITHUB_TOKEN=$(gh auth token)
just cli packages local-all
```

#### Test locally

To test locally, you need to install the packages into your local `~/.ivy` repo. This was done in previous steps.

```bash
just test-all
```

### Working with published dependencies

Release builds of the Besom SDK are published to Maven Central, and snapshots to GitHub Packages
as `org.virtuslab::besom-core:X.Y.Z` and `org.virtuslab::besom-<package>:A.B.C-core.X.Y`.

Language host provider is published to Maven as `pulumi-language-scala-vX.Y.Z-OS-ARCH.tar.gz`.

To use development version of the language host provider:

```bash
pulumi --logtostderr plugin install language scala $(cat version.txt) --server github://api.github.com/VirtusLab/besom
```

To use development version of the Besom SDKs add repository in your `project.scala`:

```scala
//> using repository sonatype:snapshots
```

or use a command line option:

```bash
--repository=sonatype:snapshots
```

To use development version of an example:

```bash
pulumi --logtostderr new https://github.com/VirtusLab/besom/tree/main/templates/aws
```

### Publishing a release

After all the testing is done, you can publish a release.

```bash
just publish-local-all
just test-all
```

#### Publish fresh packages locally

It is recommended to use `just power-wash` before publishing a release:

```bash
just power-wash
```

Make sure the repository is clean and there are no uncommitted changes:

```bash
git status -s
```

#### Bump Besom version (skip for `SNAPSHOT` re-release)

To bump Besom version in all `project.scala` and `version.txt` files:

```bash
export GITHUB_TOKEN=$(gh auth token)
just cli version bump X.Y.Z
```

Publish the new version of SDKs locally to test and provide fresh dependencies for scripts:

```bash
just publish-local-all
```

#### Update dependencies versions in all `project.scala` files (optional for patch versions)

This is most useful for `examples` and `templates`, and integration tests:

```bash
export GITHUB_TOKEN=$(gh auth token)
just cli version update
```

#### Update versions in all other places (skip for `SNAPSHOT`)

Manually update versions in all other places, specifically documentation and website, using find&replace.

Look for:
- **DO NOT** change historical versions in `CHANGELOG.md` by mistake
- `X.Y.Z` - the besom version
- `core.X.Y` - core part of the provider version
- update the provider versions in `README.md`

#### Create release branch

```bash
git checkout -b release/v$(cat version.txt)
```

#### Create a release draft on GitHub and publish language host

We need this done early to be able to publish the language host provider binaries.

```bash
just upsert-gh-release
```

Publish language host provider binaries:

```bash
just publish-language-plugins-all
```

Re-publish local language host provider binary (to fix local setup):

```bash
just install-language-plugin
```

#### Publish SDKs to maven

Publishing to maven requires:
- `OSSRH_USERNAME` - the Sonatype username
- `OSSRH_PASSWORD` - the Sonatype password
- `PGP_KEY_ID` - the signing key id (`gpg --list-keys` or `gpg --show-keys`)
- `PGP_PASSWORD` - the signing key passphrase

**Wait for the CI** to pass `Besom build and test / build` before proceeding.

Publish main SDK packages to Maven:

```bash
just publish-maven-all
```

#### Publish packages (optional for patch versions)

To publish critical provider package(s):

```bash
export GITHUB_TOKEN=$(gh auth token)
just clean-out
just cli packages maven aws awsx azure gcp docker kubernetes random command tls eks
just cli packages maven aws-native google-native azure-native
```

Tip: to check what packages are required for `examples` and `templates` use:

```bash
just cli version summary examples
just cli version summary templates
```

Publish all packages:

```bash
export GITHUB_TOKEN=$(gh auth token)
just clean-out cli packages maven-all
```

**Cation**: publishing to Maven Central is **irreversible**.
Tip: it's safer to publish the packages on-by-one or in batches due to how Maven Central behaves.
Note: `azure-native` publishing takes a long time (1-2 hours) it is recommended to handle it separately.

In case of any issues, you can try to resolve the issues manually at https://oss.sonatype.org/index.html#stagingRepositories.

#### Finish the release

Finish the release on GitHub manually, make sure the changelog is correct and correct git tag was created.

According to our Git branching and versioning strategy, the release branch should be merged after the tag is created.
Make sure to bump the git tag because GitHub Release probably already created the tag.

```bash
git push --set-upstream origin release/v$(cat version.txt)
git tag -f v$(cat version.txt)
git push -f origin v$(cat version.txt)
```

Make sure to **DO NOT squash** but merge `release/vX.Y.Z` branch into `main` otherwise the tag will not propagate to main.
Make sure to **DO NOT (auto) delete** the release branch on merge, but on the other hand, let the `release/vX.Y.Z-SNAPSHOT` be deleted on
merge to not pollute git.

#### After the release

After the release, you can bump the version to the next `-SNAPSHOT` version:

```bash
just cli version bump X.Y.Z-SNAPSHOT
```

Remember to release the snapshots to maven.

### Testing examples locally

Every example is a valid Pulumi program that can be tested by manually
doing `pulumi up` in the right folder.

```bash
cd examples/<example-name>
pulumi up
```

Here is a `just` helper to run the automated testing:

```bash
just test-example aws-webserver
```

### Testing templates locally

See `templates/README.md` on how to manually test template changes locally.

Similarly to examples, Just helper targets are provided to
automatically test templates, for example to test
`templates/default` run:

```bash
just test-template default
```

### Testing codegen locally

Codegen is tested by running:

```bash
just test-integration-codegen
```

To update test schema files from upstream:

```bash
just copy-test-schemas
```

### Testing Protobuf/gRPC codegen locally

Protobuf/gRPC codegen is tested by running:

```bash
just cli proto all
just compile-core
```

## Setting up the code editor

Both IDEs support rely on BSP and is experimental.

### BSP setup with `scala-compose`

Build experimental `scala-compose` and place on `$PATH`:

```
git clone git@github.com:VirtusLab/scala-compose.git
cd scala-compose
cat << EOF | git apply
> diff --git a/project/publish.sc b/project/publish.sc
> index e00f81ca..619d4c99 100644
> --- a/project/publish.sc
> +++ b/project/publish.sc
> @@ -113,8 +113,9 @@ def finalPublishVersion = {
>      }
>    else
>      T {
> -      val state = VcsVersion.vcsState()
> -      computePublishVersion(state, simple = true)
> +      // val state = VcsVersion.vcsState()
> +      // computePublishVersion(state, simple = true)
> +      "1.0.4"
>      }
>  }
>
> EOF
./mill -i show scala-compose.nativeImage
cp out/scala-compose/base-image/nativeImage.dest/scala-cli ~/bin/scala-compose
```

Use `scala-compose` in `besom` directory:

```bash
scala-compose setup-ide --conf-dir .
```

### IntelliJ setup

IntelliJ support is experimental.

1. Make sure you have the latest IntelliJ
2. Install Scala plugin and set update chanel to "Nightly Builds"
3. Use [BSP with `scala-cli`](https://scala-cli.virtuslab.org/docs/cookbooks/intellij-multi-bsp) (also
   see [IntelliJ documentation](https://www.jetbrains.com/help/idea/bsp-support.html))

To make sure you have `.bsp` directories, by running:

```bash
just setup-intellij
```

Now open the project in IntelliJ. If neede and add modules manually using "Project Structure > Import Module" dialog.

Additionally, please set `scalafmt` as the formatter.

### VSCode setup

If you are using VSCode:

1. Install [Metals](https://scalameta.org/metals/docs/editors/vscode#installation)
2. Open the project in Metals.

Make sure you have `.bsp` directory before you open the project in VSCode.

This might not be enough if your infrastructure is just a part (a module) of your existing Scala project.
For this to work you have to make your build tool aware of the infrastructure code,
for **sbt** create a corresponding module:

   ```scala
lazy val infra = project.in(file("infrastructure")).settings(
  libraryDependencies ++= Seq(
    "org.virtuslab" %% "besom-kubernetes" % "0.1.0", // or any other sdk you are using
    "org.virtuslab" %% "besom-core" % "0.1.0"
  ))
   ```

This just informs your IDE about the existence of the infrastructure module,
DO NOT remove dependencies from `project.scala`, because they are necessary in both places.

## Troubleshooting

If you suspect the issue is related to serialization, try to skip the preview (dry run is known to be problematic):

```bash
pulumi up --skip-preview
```

### GitHub might be throttling your requests

If you see an error like this:

- `git` failed to clone or checkout the repository
- `pulumi` failed to download the provider (401)
  GitHub might be throttling your requests, try to authenticate:

```bash
export GITHUB_TOKEN=$(gh auth token)
```

### Verbosity and debugging options

Pulumi has a few options that can help with debugging.

#### CLI command line

You can pass [debug options](https://www.pulumi.com/docs/support/troubleshooting/#verbose-logging)
to any `pulumi` CLI command, e.g.:

```bash
pulumi up -v9 --logtostderr 2> log.txt
less -R log.txt
```

Use the flag `--logflow` to apply the same log level to resource providers (but not a language provider).

#### `Pulumi.yaml` `runtime.options`

You can set `runtime.options` in `Pulumi.yaml` to pass options to the language host provider, e.g.:

```yaml
name: example
runtime:
  name: scala
  options:
    logtostderr: true
    v: 5
```

#### Environment variables

- `PULUMI_BESOM_LOG_LEVEL` - for setting Besom log level (default is `WARN`)
- `PULUMI_ENABLE_TRACE_LOGGING_TO_FILE` - for enabling Besom trace logging to file
- `PULUMI_DEBUG_COMMANDS=1` - for activating hidden debugging Pulumi CLI commands
- `TF_LOG=TRACE` - for debugging Terraform-based provider
- `PULUMI_SKIP_UPDATE_CHECK=true` - to skip Pulumi update check

More environment variables can be found in [Pulumi documentation](https://www.pulumi.com/docs/cli/environment-variables/).

#### Tracing

To collect and view [a trace](https://www.pulumi.com/docs/support/troubleshooting/#tracing):

```bash
pulumi up --tracing=file:./up.trace
PULUMI_DEBUG_COMMANDS=1 pulumi view-trace ./up.trace
```

### Downgrading Pulumi on Mac OS

As a workaround one can downgrade `pulumi` to a version, e.g. `3.94.2` using
a [commit hash](https://github.com/Homebrew/homebrew-core/commits/master/Formula/p/pulumi.rb):

```
curl -L -O https://raw.githubusercontent.com/Homebrew/homebrew-core/69b97f26bc78cf68eb30eedd0ca874b6e1914b19/Formula/p/pulumi.rb
brew install pulumi.rb
rm pulumi.rb
```

### Compilation issues

Remove `.scala-build`, e.g.:

```bash
rm -rf core/.scala-build
```

To restart `bloop` compilation server:

```bash
scala-cli bloop exit
```

To clean the builds:

```bash
just clean-all
```

If a deep cleaning needed:

```bash
just power-wash
````

To set `bloop` verbosity:

```bash
scala-cli setup-ide -v -v -v .
```

To use a nightly version of Scala compiler:

```bash
scala-cli compile -S 3.nightly .
```

To increase Scala compiler verbosity:

```bash
scala-cli compile --scalac-option -verbose .
```

To inspect a running JVM byt its PID use `jcmd`, e.g.:

```bash
jcmd 25776 VM.flags
jcmd 25776 GC.heap_info
```

## Implementation details

### Serde - gRPC/Protobuf serialization and deserialization

The most important information about our serde:

- unknown values are neither empty nor non-empty - we simply don't know
- maps (structs) don't preserve `Null` protobuf value (but we do preserve unknown values)
- upstream uses special structures signatures [besom.internal.Constants.SpecialSig](core/src/main/scala/besom/internal/codecs.scala)
  to encode internal metadata that Pulumi uses

Serialization main entry points:

- [besom.internal.RegistersOutputs](core/src/main/scala/besom/internal/RegistersOutputs.scala)
- [besom.internal.PropertiesSerializer](core/src/main/scala/besom/internal/PropertiesSerializer.scala)
- [besom.internal.Encoder](core/src/main/scala/besom/internal/codecs.scala)

Deserialization main entry points:

- [besom.internal.ResourceDecoder](core/src/main/scala/besom/internal/ResourceDecoder.scala)
- [besom.internal.Decoder](core/src/main/scala/besom/internal/codecs.scala)

Other important files:

- [besom.internal.CodecMacros](core/src/main/scala/besom/internal/CodecMacros.scala)
- [besom.internal.ResourceOps#invokeInternal](core/src/main/scala/besom/internal/ResourceOps.scala)
- [besom.internal.ProtobufUtil](core/src/main/scala/besom/internal/ProtobufUtil.scala)

## Getting Help

We are sure there are rough edges, and we appreciate you helping out.
If you want to reach out to other folks in the Besom community
please go to GitHub via [Issues](https://github.com/VirtusLab/besom/issues).
