Commit Graph

208 Commits

Author SHA1 Message Date
Tim Möhlmann
250f2344c8
feat(cache): redis cache (#8822)
# Which Problems Are Solved

Add a cache implementation using Redis single mode. This does not add
support for Redis Cluster or sentinel.

# How the Problems Are Solved

Added the `internal/cache/redis` package. All operations occur
atomically, including setting of secondary indexes, using LUA scripts
where needed.

The [`miniredis`](https://github.com/alicebob/miniredis) package is used
to run unit tests.

# Additional Changes

- Move connector code to `internal/cache/connector/...` and remove
duplicate code from `query` and `command` packages.
- Fix a missed invalidation on the restrictions projection

# Additional Context

Closes #8130
2024-11-04 10:44:51 +00:00
Fabi
3c4a92a981
chore(github): add type to issue templates (#8775)
# Which Problems Are Solved

Github introduced the new issue types, which we want to add to our
issues. Starting point is to change the templates, so we can add the
right types.
2024-10-15 09:42:16 +00:00
Fabi
222915ca3d
chore: test issue type (#8750)
# Which Problems Are Solved

We want to use the new issue types of github
2024-10-09 14:03:40 +02:00
Tim Möhlmann
d8a71d217c
test: upload integration server logs as artifacts (#8600)
# Which Problems Are Solved

Upload the integration test server logs as artifacts, even if the tests
fail.
Before this change logs where printed through the Makefile.
However if a test would fail, the logs wouldn't get printed.

# How the Problems Are Solved

- Add an extra build step that pushes `tmp/zitadel.log` and
`tmp/race.log.$pid` to artificats storage.
- Logs are no longer printed in the `core_integration_reports` Makefile
recipe.

# Additional Changes

Do not remove coverage data when generating the coverage report in
`core_integration_reports`. This is to prevent future "File not found"
erros when running the command repeatedly.

# Additional Context

Reported as internal feedback
2024-09-11 17:51:18 +00:00
Tim Möhlmann
d2e0ac07f1
chore(tests): use a coverage server binary (#8407)
# Which Problems Are Solved

Use a single server instance for API integration tests. This optimizes
the time taken for the integration test pipeline,
because it allows running tests on multiple packages in parallel. Also,
it saves time by not start and stopping a zitadel server for every
package.

# How the Problems Are Solved

- Build a binary with `go build -race -cover ....`
- Integration tests only construct clients. The server remains running
in the background.
- The integration package and tested packages now fully utilize the API.
No more direct database access trough `query` and `command` packages.
- Use Makefile recipes to setup, start and stop the server in the
background.
- The binary has the race detector enabled
- Init and setup jobs are configured to halt immediately on race
condition
- Because the server runs in the background, races are only logged. When
the server is stopped and race logs exist, the Makefile recipe will
throw an error and print the logs.
- Makefile recipes include logic to print logs and convert coverage
reports after the server is stopped.
- Some tests need a downstream HTTP server to make requests, like quota
and milestones. A new `integration/sink` package creates an HTTP server
and uses websockets to forward HTTP request back to the test packages.
The package API uses Go channels for abstraction and easy usage.

# Additional Changes

- Integration test files already used the `//go:build integration`
directive. In order to properly split integration from unit tests,
integration test files need to be in a `integration_test` subdirectory
of their package.
- `UseIsolatedInstance` used to overwrite the `Tester.Client` for each
instance. Now a `Instance` object is returned with a gRPC client that is
connected to the isolated instance's hostname.
- The `Tester` type is now `Instance`. The object is created for the
first instance, used by default in any test. Isolated instances are also
`Instance` objects and therefore benefit from the same methods and
values. The first instance and any other us capable of creating an
isolated instance over the system API.
- All test packages run in an Isolated instance by calling
`NewInstance()`
- Individual tests that use an isolated instance use `t.Parallel()`

# Additional Context

- Closes #6684
- https://go.dev/doc/articles/race_detector
- https://go.dev/doc/build-cover

---------

Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2024-09-06 14:47:57 +02:00
Elio Bischof
9a1140289a
chore: automatically bump charts (#8537)
# Which Problems Are Solved

Bumping charts needs a manual trigger.

# How the Problems Are Solved

The charts bump workflow is run after every ZITADEL release.

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-09-04 05:16:17 +00:00
Elio Bischof
fdf0434133
fix(console): remove navigation flakiness (#8439)
# Which Problems Are Solved

The navigation in the console default settings is flaky. Sometimes it
arbitrarily jumps to the organizations page.

# How the Problems Are Solved

The lifecycle hooks were extended to react differently to changes that
come from 'outside' and from the component itself.

# Additional Changes

The e2e tests are supposed to run against Firefox and Chrome. However
they are run twice against Electon. Fixing this revealed the console
navigation flakiness that was less visible on Electron.

The following issues are also fixed with this PR to reduce flakiness in
e2e tests.

- The custom command in the pipeline is removed from the e2e action
step, so the browser argument is respected.
- The npm packages of the e2e tests are updated to their latest version.
- Notification tests run against a clean state now so they don't depend
on each other anymore. This resolved some flakiness and improved
debuggability of the tests.
- E2E page load timeout is increased, reducing flakiness.
- E2E tests wait on some elements to be enabled before they interact
with them, reducing flakiness.

# Additional Context

- Closes #8404 
- Follow-up: https://github.com/zitadel/zitadel/issues/8471

The e2e tests ran three times in a row successfully in the pipeline
against both browsers.

---------

Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: Livio Spring <livio.a@gmail.com>
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
2024-08-22 09:48:36 +02:00
Fabi
2e7235ebf2
fix: change pr template to not link to existing issues and prs (#8412)
# Which Problems Are Solved

In the PR template we have added some ideas about additional context,
but we link to existing prs and issues as an example.
So everytime someone doesn't change the description when creating the
issue, its a mention to that issue or pr.


# How the Problems Are Solved
replace with non existing values


![Uploading image.png…]()
2024-08-09 15:41:40 +02:00
Elio Bischof
d95c9508a3
chore(lint): print error locations (#8246)
# Which Problems Are Solved

If golangci-lint fails, it is hard to find out how to fix it.

# How the Problems Are Solved

The official GitHub action for golangci-lint is upgraded from v4 to
[v6](https://github.com/golangci/golangci-lint-action/releases/tag/v6.0.1).

## Before Upgrade

![image](https://github.com/zitadel/zitadel/assets/12727842/d25bcb0d-5e86-4f8d-92e9-f24e53319062)

## After Upgrade

![image](https://github.com/zitadel/zitadel/assets/12727842/91c6c9ff-832b-49f5-ba92-ff835d8179c0)

# Additional Changes

Removes the config properties `skip-pkg-cache` and `skip-build-cache` as
support was removed with
[v5](https://github.com/golangci/golangci-lint-action/releases/tag/v5.0.0).

---------

Co-authored-by: Silvan <silvan.reusser@gmail.com>
2024-07-17 07:20:45 +00:00
Florian Forster
cf72bd6192
chore(docs): use vercel ci again because GH secret problem on forks (#8142)
# Fallback to Vercel CI

Since we cannot share the vercel_token on forks we cannot deploy by
vercel CLI.
This PR reverts to the last working state by using vercel CI.

I will look into a fix with an npm script or a turbo config to ignore
builds on folder changes.
2024-06-14 16:50:58 +02:00
Florian Forster
21ffe4f693
chore(docs): fix ci (#8135)
# Fixing Fork Previews

This PR tries to fix the upload of the docs preview to vercel which was
not working due to the not available secret.
2024-06-14 13:50:36 +00:00
Florian Forster
f6a50db96c
chore(docs): fix prod build for vercel (#8121)
# Fix

Fixes a problem on main
2024-06-13 08:47:10 +00:00
Florian Forster
c9e352033e
chore(docs): use gh action to build docs (#8097)
# Which Problems Are Solved

This allows us to build multiple docs in parallel and only runs when
docs/proto are changed.

# Additional Changes

- [ ] Change "required" in GitHub from Vercel to the docs flow

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-06-13 08:07:39 +00:00
Elio Bischof
eefa35715f
chore: tag author in ready for review comment (#8009)
# Which Problems Are Solved

It is not very clear if the author or the reviewer of a PR should tick
the boxes.

# How the Problems Are Solved

The author of the PR is tagged in the comment, because the author should
tick the boxes before marking it as ready for review.
2024-05-24 18:45:06 +02:00
Silvan
0bfcf2c317
refactor(query): use new packages for org by id query (#7826)
If the feature is enabled the new packages are used to query org by id

Part of: https://github.com/zitadel/zitadel/issues/7639

### Definition of Ready

- [x] I am happy with the code
- [x] Short description of the feature/issue is added in the pr
description
- [x] PR is linked to the corresponding user story
- [ ] Acceptance criteria are met
- [ ] All open todos and follow ups are defined in a new ticket and
justified
- [ ] Deviations from the acceptance criteria and design are agreed with
the PO and documented.
- [x] No debug or dead code
- [x] My code has no repetitions
- [ ] Critical parts are tested automatically
- [ ] Where possible E2E tests are implemented
- [ ] Documentation/examples are up-to-date
- [ ] All non-functional requirements are met
- [x] Functionality of the acceptance criteria is checked manually on
the dev system.
2024-05-24 13:32:57 +02:00
Elio Bischof
3396657c3f
chore: improve commit messages (#7921)
* chore: improve commit messages

* indent

* cleanup

* additional changes

---------

Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
2024-05-09 20:12:42 +03:00
Stefan Benz
87e4a0be20
chore: build and test on main for codecov (#7875) 2024-04-30 20:08:00 +03:00
Stefan Benz
4520c6fc49
chore: codecov token secret for nested workflow (#7792)
fix: codecov token secret for nested workflow
2024-04-22 13:10:49 +00:00
dependabot[bot]
6f16561f4d
chore(deps): bump actions/add-to-project from 0.6.1 to 1.0.1 (#7767)
Bumps [actions/add-to-project](https://github.com/actions/add-to-project) from 0.6.1 to 1.0.1.
- [Release notes](https://github.com/actions/add-to-project/releases)
- [Commits](https://github.com/actions/add-to-project/compare/v0.6.1...v1.0.1)

---
updated-dependencies:
- dependency-name: actions/add-to-project
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-04-15 10:36:23 +00:00
dependabot[bot]
6a1ec149d3
chore(deps): bump actions/github-script from 6 to 7 (#7768)
Bumps [actions/github-script](https://github.com/actions/github-script) from 6 to 7.
- [Release notes](https://github.com/actions/github-script/releases)
- [Commits](https://github.com/actions/github-script/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/github-script
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-04-15 10:13:19 +00:00
dependabot[bot]
e613ad6453
chore(deps): bump codecov/codecov-action from 4.1.0 to 4.3.0 (#7747)
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 4.1.0 to 4.3.0.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v4.1.0...v4.3.0)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-04-15 09:50:47 +00:00
Tim Möhlmann
fe9bb49caa
chore(deps): update all go deps (#7773)
This change updates all go modules, including oidc, a major version of go-jose and the go 1.22 release.
2024-04-15 09:17:36 +00:00
Elio Bischof
0267415126
chore: avoid latest on maintenance releases (#7702)
* chore: avoid latest on maintenance releases

* update semantic

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-04-10 11:56:05 +00:00
Livio Spring
f862e43ede
chore(workflow): run e2e on non standard runners (#7698)
Co-authored-by: Elio Bischof <elio@zitadel.com>
2024-04-03 22:15:00 +00:00
Livio Spring
4c945f8bdc
chore(workflow): stop previous GH action pipeline on new changes and move back to public runners (#7659)
* chore(workflow): stop previous GH action pipeline on new changes

* skip previous code ql actions

* try running e2e on gh runner again
2024-03-28 07:15:03 +00:00
dependabot[bot]
47e5533f0f
chore(deps): bump actions/add-to-project from 0.6.0 to 0.6.1 (#7628)
Bumps [actions/add-to-project](https://github.com/actions/add-to-project) from 0.6.0 to 0.6.1.
- [Release notes](https://github.com/actions/add-to-project/releases)
- [Commits](https://github.com/actions/add-to-project/compare/v0.6.0...v0.6.1)

---
updated-dependencies:
- dependency-name: actions/add-to-project
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-03-25 09:07:57 +00:00
Livio Spring
2bcc42c4cd
chore(workflow): fix homebrew update (#7630) 2024-03-25 09:29:54 +01:00
Silvan
5b301c7f96
chore: trigger update homebrew tap on latest release (#7618) 2024-03-22 13:38:18 +01:00
Livio Spring
bbb4dea0bd
chore: fix artifact upload for container build (#7565) 2024-03-14 11:10:53 +00:00
Livio Spring
5d2cfc06d5
chore: update github actions (#7564) 2024-03-14 09:26:31 +00:00
Silvan
4224c7ad3a
chore(core): update dependencies (#7517)
* chore(core): update dependencies

* chore(core): update dependencies
2024-03-11 10:43:56 +00:00
dependabot[bot]
213c425806
chore(deps): bump codecov/codecov-action from 3.1.4 to 4.1.0 (#7470)
* chore(deps): bump codecov/codecov-action from 3.1.4 to 4.1.0

Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 3.1.4 to 4.1.0.
- [Release notes](https://github.com/codecov/codecov-action/releases)
- [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/codecov/codecov-action/compare/v3.1.4...v4.1.0)

---
updated-dependencies:
- dependency-name: codecov/codecov-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* add CODECOV_TOKEN

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-03-08 08:20:33 +01:00
dependabot[bot]
87086c190b
chore(deps): bump docker/build-push-action from 4 to 5 (#7469)
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 4 to 5.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v4...v5)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-08 06:52:49 +00:00
dependabot[bot]
37eea6940a
chore(deps): bump actions/cache from 3 to 4 (#7473)
Bumps [actions/cache](https://github.com/actions/cache) from 3 to 4.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-08 07:38:32 +01:00
Livio Spring
98bd355ce2
chore(workflow): update github actions (#7508) 2024-03-07 15:38:38 +00:00
dependabot[bot]
c03c054aea
chore(deps): bump docker/setup-qemu-action from 2 to 3 (#7474)
Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 2 to 3.
- [Release notes](https://github.com/docker/setup-qemu-action/releases)
- [Commits](https://github.com/docker/setup-qemu-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: docker/setup-qemu-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-06 08:29:51 +00:00
dependabot[bot]
a8dbb052de
chore(deps): bump actions/upload-artifact from 3 to 4 (#7471)
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3 to 4.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-06 07:26:03 +00:00
Fabi
437c834a40
chore: update github action with correct dependabot actor (#7485) 2024-03-01 07:10:14 +00:00
Livio Spring
60a73df459
chore: add github-actions to dependabot (#7468) 2024-02-29 09:17:38 +00:00
Elio Bischof
585988bd83
chore(ci): merge test runs (#7388)
* chore(ci): merge test runs

* docs: improve CONTRIBUTING.md

* expect failure

* expect failure unit

* cleanup

* fix: correct binding in integration tests

* fix: correct binding in integration tests

---------

Co-authored-by: Stefan Benz <stefan@caos.ch>
2024-02-19 07:50:37 +01:00
Livio Spring
00d245eaf3
chore(codeql): use go version from go.mod (#6807)
* chore(codeql): use go version from go.mod

* try 2.15.0 explicitly

* tag version

* Update codeql.yml

* Update codeql.yml

* try make

* Update codeql.yml

* Update codeql.yml

* Update codeql.yml

* Update codeql.yml

* Update codeql.yml

* Update codeql.yml

* Update codeql.yml

* Update codeql.yml

* Update codeql.yml

* Update codeql.yml

* Update codeql.yml

* Update codeql.yml

* Update codeql.yml

* Update codeql.yml

* Update codeql.yml

* use make compile and remove matrix check

* Update codeql.yml

* Update codeql.yml

* retry autobuild

* Update codeql.yml

* try autobuild with generated stub

* chore: change init step for codeQL

* chore: codeql

* chore: try new codeql version

* chore: new codeql-action version

* chore: new codeql-action version

* chore: new codeql-action version

* chore: new codeql-action version

---------

Co-authored-by: Stefan Benz <stefan@caos.ch>
2024-02-16 10:54:41 +00:00
Fabi
e699103303
chore: update pm board action (#7330)
automatically ad prs of non engineers to board and label community prs
2024-02-02 10:59:37 +00:00
Livio Spring
8470649ecb
chore: pin crdb version for unit tests (#7260)
* chore: pin crdb version for unit tests

* use latest 23.1 version

* use latest available 23.1 version
2024-01-18 08:16:54 +00:00
Silvan
4e3936b5bf
ci: publish sha to docker registry (#7127) 2024-01-02 14:03:23 +00:00
Tim Möhlmann
115d944d38
chore: add database questions to bug report template (#6975)
* chore: add database questions to bug report template

Sometimes we get bug reports that are only reproducible when zitadel is running against a certain database.
This change adds database related questions to the issue template, as it is something people don't tend to describe in the detail fields.

* fix syntax error
2023-11-27 08:23:49 +01:00
mffap
8c20548db7
chore: update bug template (#6924)
Provide link to the product management in the issue's markdown instead of description. This allows people to actually follow the link when creating an issue.
2023-11-23 10:09:40 +00:00
Tim Möhlmann
ba9b807854
perf(oidc): optimize the introspection endpoint (#6909)
* get key by id and cache them

* userinfo from events for v2 tokens

* improve keyset caching

* concurrent token and client checks

* client and project in single query

* logging and otel

* drop owner_removed column on apps and authN tables

* userinfo and project roles in go routines

* get  oidc user info from projections and add actions

* add avatar URL

* some cleanup

* pull oidc work branch

* remove storage from server

* add config flag for experimental introspection

* legacy introspection flag

* drop owner_removed column on user projections

* drop owner_removed column on useer_metadata

* query userinfo unit test

* query introspection client test

* add user_grants to the userinfo query

* handle PAT scopes

* bring triggers back

* test instance keys query

* add userinfo unit tests

* unit test keys

* go mod tidy

* solve some bugs

* fix missing preferred login name

* do not run triggers in go routines, they seem to deadlock

* initialize the trigger handlers late with a sync.OnceValue

* Revert "do not run triggers in go routines, they seem to deadlock"

This reverts commit 2a03da2127.

* add missing translations

* chore: update go version for linting

* pin oidc version

* parse a global time location for query test

* fix linter complains

* upgrade go lint

* fix more linting issues

---------

Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2023-11-21 13:11:38 +01:00
Silvan
0948a0b9ae
ci: set runner group on parallel jobs (#6916) 2023-11-14 10:48:41 +00:00
adlerhurst
bd23a7a56f merge main into next 2023-10-19 12:34:00 +02:00
adlerhurst
9c069806c4 ci(lint): use ref_base instead of main 2023-10-19 10:47:37 +02:00