mirror of
https://github.com/portainer/portainer.git
synced 2025-07-24 15:59:41 +02:00
chore(store) EE-1981: Refactor/store/error checking, and other refactoring (#6173)
* use the Store interface IsErrObjectNotFound() to avoid revealing internal errors Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * what happens when you extract the datastore interfaces into their own package Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * Start renaming Storage methods Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * extract the boltdb specific code from the Portainer storage code (example, the others need the same) Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * more extract bolt.Tx from datastore code Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * minimise imports by putting moving the struct definition into the file that needs the Service imports Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * more extraction of boltdb.Tx Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * extract the use of bucket.SetSequence Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * almost done - just endpoint.Synchonise :/ Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * so, endpoint.Synchonize looks hard, but i can't find where we use it, so 'delete first refactoring' Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * fix test compile errors Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * test compile fixes after rebase Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * fix a mis-remembering I had wrt deserialisation - last time i used AnyData - jsoniter's bindTo looks interesting for the same reason Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * set us up to make the connection an interface Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * make the db connection a datastore interface, and separate out our datastore services from the bolt ones Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * rename methods to something less oltdb internals specific Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * these errors are not boltdb secific Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * start using the db-backend factory method too Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * export boltdb raw in case we can't export from the service layer Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * add a raw export from boltdb to yaml for broken db's, and an export services to yaml in backup Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * add the version info by hand for now Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * actually, the export from services can be fully typed - its the import that needs to do more work Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * redo raw export, and make import capable of using it Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * add DockerHub Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * migration from anything older than v1.21.0 has been broken for quite a while, deleting the un-tested code Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * fix go test ./... again Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * my goland wasn't setup to gofmt Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * move the two extremely dubious migration tests down into store, so they can use the test store code Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * the migrator is now free of boltdb Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * reverse goland overzealous replcement of internal with boltdb Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * more undo over-zealous goland internal->boltdb Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * yay, now bolt is only mentioned inside the api/database/ dir Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * and this might be the last of the boltdb references? Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * add todo Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * extract the store code into a separate module too Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * don't need the fileService in boltdb anymore Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * use IsErrObjectNotFound() Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * use a string to select what database backend we use Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * make isNew store an ephemeral bool that doesn't stay true after we've initialised it Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * move the import.json wip to a separate file so its more obvious - we'll be using it for testing, emergency fixups, and in the next part of the store work, when we improve migrations and data model lifecycles Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * undo vscode formatting html Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * fix app templates symbol (#6221) * feat(webhook) EE-2125 send registry auth haeder when update swarms service via webhook (#6220) * feat(webhook) EE-2125 add some helpers to registry utils * feat(webhook) EE-2125 persist registryID when creating a webhook * feat(webhook) EE-2125 send registry auth header when executing a webhook * feat(webhook) EE-2125 send registryID to backend when creating a service with webhook * feat(webhook) EE-2125 use the initial registry ID to create webhook on editing service screen * feat(webhook) EE-2125 update webhook when update registry * feat(webhook) EE-2125 add endpoint of update webhook * feat(webhook) EE-2125 code cleanup * feat(webhook) EE-2125 fix a typo * feat(webhook) EE-2125 fix circle import issue with unit test Co-authored-by: Simon Meng <simon.meng@portainer.io> * fix(kubeconfig): show kubeconfig download button for non admin users [EE-2123] (#6204) Co-authored-by: Simon Meng <simon.meng@portainer.io> * fix data-cy for k8s cluster menu (#6226) LGTM * feat(stack): make stack created from app template editable EE-1941 (#6104) feat(stack): make stack from app template editable * fix(container):disable Duplicate/Edit button when the container is portainer (#6223) * fix/ee-1909/show-pull-image-error (#6195) Co-authored-by: sunportainer <ericsun@SG1.local> * feat(cy): add data-cy to helm install button (#6241) * feat(cy): add data-cy to add registry button (#6242) * refactor(app): convert root folder files to es6 (#4159) * refactor(app): duplicate constants as es6 exports (#4158) * fix(docker): provide workaround to save network name variable (#6080) * fix/EE-1862/unable-to-stop-or-remove-stack workaround for var without default value in yaml file * fix/EE-1862/unable-to-stop-or-remove-stack check yaml file * fixed func and var names * wrapper error and used bool for stringset * UT case for createNetworkEnvFile * UT case for %s=%s * powerful StringSet * wrapper error for extract network name * wrapper all the return err * store more env * put to env file * make default value None * feat: gzip static resources (#6258) * fix(ssl)//handle --sslcert and --sslkey ee-2106 (#6203) * fix/ee-2106/handle-sslcert-sslkey Co-authored-by: sunportainer <ericsun@SG1.local> * fix(server):support disable https only ee-2068 (#6232) * fix/ee-2068/disable-forcely-https * feat(store): implement store tests EE-2112 (#6224) * add store tests * add some more tests * Update missing helm user repo methods * remove redundant comments * add webhook export * update webhooks * use the Store interface IsErrObjectNotFound() to avoid revealing internal errors Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * what happens when you extract the datastore interfaces into their own package Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * Start renaming Storage methods Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * extract the boltdb specific code from the Portainer storage code (example, the others need the same) Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * more extract bolt.Tx from datastore code Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * minimise imports by putting moving the struct definition into the file that needs the Service imports Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * more extraction of boltdb.Tx Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * extract the use of bucket.SetSequence Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * almost done - just endpoint.Synchonise :/ Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * so, endpoint.Synchonize looks hard, but i can't find where we use it, so 'delete first refactoring' Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * fix test compile errors Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * test compile fixes after rebase Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * fix a mis-remembering I had wrt deserialisation - last time i used AnyData - jsoniter's bindTo looks interesting for the same reason Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * set us up to make the connection an interface Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * make the db connection a datastore interface, and separate out our datastore services from the bolt ones Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * rename methods to something less oltdb internals specific Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * these errors are not boltdb secific Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * start using the db-backend factory method too Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * export boltdb raw in case we can't export from the service layer Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * add a raw export from boltdb to yaml for broken db's, and an export services to yaml in backup Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * add the version info by hand for now Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * actually, the export from services can be fully typed - its the import that needs to do more work Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * redo raw export, and make import capable of using it Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * add DockerHub Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * migration from anything older than v1.21.0 has been broken for quite a while, deleting the un-tested code Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * fix go test ./... again Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * my goland wasn't setup to gofmt Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * move the two extremely dubious migration tests down into store, so they can use the test store code Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * the migrator is now free of boltdb Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * reverse goland overzealous replcement of internal with boltdb Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * more undo over-zealous goland internal->boltdb Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * yay, now bolt is only mentioned inside the api/database/ dir Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * and this might be the last of the boltdb references? Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * add todo Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * extract the store code into a separate module too Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * don't need the fileService in boltdb anymore Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * use IsErrObjectNotFound() Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * use a string to select what database backend we use Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * make isNew store an ephemeral bool that doesn't stay true after we've initialised it Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * move the import.json wip to a separate file so its more obvious - we'll be using it for testing, emergency fixups, and in the next part of the store work, when we improve migrations and data model lifecycles Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * undo vscode formatting html Signed-off-by: Sven Dowideit <sven.dowideit@portainer.io> * Update missing helm user repo methods * feat(store): implement store tests EE-2112 (#6224) * add store tests * add some more tests * remove redundant comments * add webhook export * update webhooks * fix build issues after rebasing * move migratorparams * remove unneeded integer type conversions * disable the db import/export for now Co-authored-by: Richard Wei <54336863+WaysonWei@users.noreply.github.com> Co-authored-by: cong meng <mcpacino@gmail.com> Co-authored-by: Simon Meng <simon.meng@portainer.io> Co-authored-by: Marcelo Rydel <marcelorydel26@gmail.com> Co-authored-by: Hao Zhang <hao.zhang@portainer.io> Co-authored-by: sunportainer <93502624+sunportainer@users.noreply.github.com> Co-authored-by: sunportainer <ericsun@SG1.local> Co-authored-by: wheresolivia <78844659+wheresolivia@users.noreply.github.com> Co-authored-by: Chaim Lev-Ari <chiptus@users.noreply.github.com> Co-authored-by: Chao Geng <93526589+chaogeng77977@users.noreply.github.com> Co-authored-by: Dmitry Salakhov <to@dimasalakhov.com> Co-authored-by: Matt Hook <hookenz@gmail.com>
This commit is contained in:
Sven Dowideit
GitHub
parent
b02bf0c9d7
commit
f99329eb7e
299 changed files with 5246 additions and 5092 deletions
194
api/datastore/migrator/migrate_ce.go
Normal file
194
api/datastore/migrator/migrate_ce.go
Normal file
|
|
@ -0,0 +1,194 @@
|
|||
package migrator
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
werrors "github.com/pkg/errors"
|
||||
portainer "github.com/portainer/portainer/api"
|
||||
)
|
||||
|
||||
func migrationError(err error, context string) error {
|
||||
return werrors.Wrap(err, "failed in "+context)
|
||||
}
|
||||
|
||||
// Migrate checks the database version and migrate the existing data to the most recent data model.
|
||||
func (m *Migrator) Migrate() error {
|
||||
// set DB to updating status
|
||||
err := m.versionService.StoreIsUpdating(true)
|
||||
if err != nil {
|
||||
return migrationError(err, "StoreIsUpdating")
|
||||
}
|
||||
|
||||
if m.currentDBVersion < 17 {
|
||||
return migrationError(err, "migrating from less than Portainer 1.21.0 is not supported, please contact Portainer support.")
|
||||
}
|
||||
|
||||
// Portainer 1.21.0
|
||||
if m.currentDBVersion < 18 {
|
||||
err := m.updateUsersToDBVersion18()
|
||||
if err != nil {
|
||||
return migrationError(err, "updateUsersToDBVersion18")
|
||||
}
|
||||
|
||||
err = m.updateEndpointsToDBVersion18()
|
||||
if err != nil {
|
||||
return migrationError(err, "updateEndpointsToDBVersion18")
|
||||
}
|
||||
|
||||
err = m.updateEndpointGroupsToDBVersion18()
|
||||
if err != nil {
|
||||
return migrationError(err, "updateEndpointGroupsToDBVersion18")
|
||||
}
|
||||
|
||||
err = m.updateRegistriesToDBVersion18()
|
||||
if err != nil {
|
||||
return migrationError(err, "updateRegistriesToDBVersion18")
|
||||
}
|
||||
}
|
||||
|
||||
// Portainer 1.22.0
|
||||
if m.currentDBVersion < 19 {
|
||||
err := m.updateSettingsToDBVersion19()
|
||||
if err != nil {
|
||||
return migrationError(err, "updateSettingsToDBVersion19")
|
||||
}
|
||||
}
|
||||
|
||||
// Portainer 1.22.1
|
||||
if m.currentDBVersion < 20 {
|
||||
err := m.updateUsersToDBVersion20()
|
||||
if err != nil {
|
||||
return migrationError(err, "updateUsersToDBVersion20")
|
||||
}
|
||||
|
||||
err = m.updateSettingsToDBVersion20()
|
||||
if err != nil {
|
||||
return migrationError(err, "updateSettingsToDBVersion20")
|
||||
}
|
||||
|
||||
err = m.updateSchedulesToDBVersion20()
|
||||
if err != nil {
|
||||
return migrationError(err, "updateSchedulesToDBVersion20")
|
||||
}
|
||||
}
|
||||
|
||||
// Portainer 1.23.0
|
||||
// DBVersion 21 is missing as it was shipped as via hotfix 1.22.2
|
||||
if m.currentDBVersion < 22 {
|
||||
err := m.updateResourceControlsToDBVersion22()
|
||||
if err != nil {
|
||||
return migrationError(err, "updateResourceControlsToDBVersion22")
|
||||
}
|
||||
|
||||
err = m.updateUsersAndRolesToDBVersion22()
|
||||
if err != nil {
|
||||
return migrationError(err, "updateUsersAndRolesToDBVersion22")
|
||||
}
|
||||
}
|
||||
|
||||
// Portainer 1.24.0
|
||||
if m.currentDBVersion < 23 {
|
||||
migrateLog.Info("Migrating to DB 23")
|
||||
err := m.updateTagsToDBVersion23()
|
||||
if err != nil {
|
||||
return migrationError(err, "updateTagsToDBVersion23")
|
||||
}
|
||||
|
||||
err = m.updateEndpointsAndEndpointGroupsToDBVersion23()
|
||||
if err != nil {
|
||||
return migrationError(err, "updateEndpointsAndEndpointGroupsToDBVersion23")
|
||||
}
|
||||
}
|
||||
|
||||
// Portainer 1.24.1
|
||||
if m.currentDBVersion < 24 {
|
||||
migrateLog.Info("Migrating to DB 24")
|
||||
err := m.updateSettingsToDB24()
|
||||
if err != nil {
|
||||
return migrationError(err, "updateSettingsToDB24")
|
||||
}
|
||||
}
|
||||
|
||||
// Portainer 2.0.0
|
||||
if m.currentDBVersion < 25 {
|
||||
migrateLog.Info("Migrating to DB 25")
|
||||
err := m.updateSettingsToDB25()
|
||||
if err != nil {
|
||||
return migrationError(err, "updateSettingsToDB25")
|
||||
}
|
||||
|
||||
err = m.updateStacksToDB24()
|
||||
if err != nil {
|
||||
return migrationError(err, "updateStacksToDB24")
|
||||
}
|
||||
}
|
||||
|
||||
// Portainer 2.1.0
|
||||
if m.currentDBVersion < 26 {
|
||||
migrateLog.Info("Migrating to DB 26")
|
||||
err := m.updateEndpointSettingsToDB25()
|
||||
if err != nil {
|
||||
return migrationError(err, "updateEndpointSettingsToDB25")
|
||||
}
|
||||
}
|
||||
|
||||
// Portainer 2.2.0
|
||||
if m.currentDBVersion < 27 {
|
||||
migrateLog.Info("Migrating to DB 27")
|
||||
err := m.updateStackResourceControlToDB27()
|
||||
if err != nil {
|
||||
return migrationError(err, "updateStackResourceControlToDB27")
|
||||
}
|
||||
}
|
||||
|
||||
// Portainer 2.6.0
|
||||
if m.currentDBVersion < 30 {
|
||||
migrateLog.Info("Migrating to DB 30")
|
||||
err := m.migrateDBVersionToDB30()
|
||||
if err != nil {
|
||||
return migrationError(err, "migrateDBVersionToDB30")
|
||||
}
|
||||
}
|
||||
|
||||
// Portainer 2.9.0
|
||||
if m.currentDBVersion < 32 {
|
||||
err := m.migrateDBVersionToDB32()
|
||||
if err != nil {
|
||||
return migrationError(err, "migrateDBVersionToDB32")
|
||||
}
|
||||
}
|
||||
|
||||
// Portainer 2.9.1, 2.9.2
|
||||
if m.currentDBVersion < 33 {
|
||||
migrateLog.Info("Migrating to DB 33")
|
||||
err := m.migrateDBVersionToDB33()
|
||||
if err != nil {
|
||||
return migrationError(err, "migrateDBVersionToDB33")
|
||||
}
|
||||
}
|
||||
|
||||
// Portainer 2.10
|
||||
if m.currentDBVersion < 34 {
|
||||
migrateLog.Info("Migrating to DB 34")
|
||||
if err := m.migrateDBVersionToDB34(); err != nil {
|
||||
return migrationError(err, "migrateDBVersionToDB34")
|
||||
}
|
||||
}
|
||||
|
||||
// Portainer 2.9.3 (yep out of order, but 2.10 is EE only)
|
||||
if m.currentDBVersion < 35 {
|
||||
migrateLog.Info("Migrating to DB 35")
|
||||
if err := m.migrateDBVersionToDB35(); err != nil {
|
||||
return migrationError(err, "migrateDBVersionToDB35")
|
||||
}
|
||||
}
|
||||
|
||||
err = m.versionService.StoreDBVersion(portainer.DBVersion)
|
||||
if err != nil {
|
||||
return migrationError(err, "StoreDBVersion")
|
||||
}
|
||||
migrateLog.Info(fmt.Sprintf("Updated DB version to %d", portainer.DBVersion))
|
||||
|
||||
// reset DB updating status
|
||||
return m.versionService.StoreIsUpdating(false)
|
||||
}
|
||||
125
api/datastore/migrator/migrate_dbversion17.go
Normal file
125
api/datastore/migrator/migrate_dbversion17.go
Normal file
|
|
@ -0,0 +1,125 @@
|
|||
package migrator
|
||||
|
||||
import (
|
||||
portainer "github.com/portainer/portainer/api"
|
||||
)
|
||||
|
||||
func (m *Migrator) updateUsersToDBVersion18() error {
|
||||
legacyUsers, err := m.userService.Users()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, user := range legacyUsers {
|
||||
user.PortainerAuthorizations = map[portainer.Authorization]bool{
|
||||
portainer.OperationPortainerDockerHubInspect: true,
|
||||
portainer.OperationPortainerEndpointGroupList: true,
|
||||
portainer.OperationPortainerEndpointList: true,
|
||||
portainer.OperationPortainerEndpointInspect: true,
|
||||
portainer.OperationPortainerEndpointExtensionAdd: true,
|
||||
portainer.OperationPortainerEndpointExtensionRemove: true,
|
||||
portainer.OperationPortainerExtensionList: true,
|
||||
portainer.OperationPortainerMOTD: true,
|
||||
portainer.OperationPortainerRegistryList: true,
|
||||
portainer.OperationPortainerRegistryInspect: true,
|
||||
portainer.OperationPortainerTeamList: true,
|
||||
portainer.OperationPortainerTemplateList: true,
|
||||
portainer.OperationPortainerTemplateInspect: true,
|
||||
portainer.OperationPortainerUserList: true,
|
||||
portainer.OperationPortainerUserMemberships: true,
|
||||
}
|
||||
|
||||
err = m.userService.UpdateUser(user.ID, &user)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *Migrator) updateEndpointsToDBVersion18() error {
|
||||
legacyEndpoints, err := m.endpointService.Endpoints()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, endpoint := range legacyEndpoints {
|
||||
endpoint.UserAccessPolicies = make(portainer.UserAccessPolicies)
|
||||
for _, userID := range endpoint.AuthorizedUsers {
|
||||
endpoint.UserAccessPolicies[userID] = portainer.AccessPolicy{
|
||||
RoleID: 4,
|
||||
}
|
||||
}
|
||||
|
||||
endpoint.TeamAccessPolicies = make(portainer.TeamAccessPolicies)
|
||||
for _, teamID := range endpoint.AuthorizedTeams {
|
||||
endpoint.TeamAccessPolicies[teamID] = portainer.AccessPolicy{
|
||||
RoleID: 4,
|
||||
}
|
||||
}
|
||||
|
||||
err = m.endpointService.UpdateEndpoint(endpoint.ID, &endpoint)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *Migrator) updateEndpointGroupsToDBVersion18() error {
|
||||
legacyEndpointGroups, err := m.endpointGroupService.EndpointGroups()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, endpointGroup := range legacyEndpointGroups {
|
||||
endpointGroup.UserAccessPolicies = make(portainer.UserAccessPolicies)
|
||||
for _, userID := range endpointGroup.AuthorizedUsers {
|
||||
endpointGroup.UserAccessPolicies[userID] = portainer.AccessPolicy{
|
||||
RoleID: 4,
|
||||
}
|
||||
}
|
||||
|
||||
endpointGroup.TeamAccessPolicies = make(portainer.TeamAccessPolicies)
|
||||
for _, teamID := range endpointGroup.AuthorizedTeams {
|
||||
endpointGroup.TeamAccessPolicies[teamID] = portainer.AccessPolicy{
|
||||
RoleID: 4,
|
||||
}
|
||||
}
|
||||
|
||||
err = m.endpointGroupService.UpdateEndpointGroup(endpointGroup.ID, &endpointGroup)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *Migrator) updateRegistriesToDBVersion18() error {
|
||||
legacyRegistries, err := m.registryService.Registries()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, registry := range legacyRegistries {
|
||||
registry.UserAccessPolicies = make(portainer.UserAccessPolicies)
|
||||
for _, userID := range registry.AuthorizedUsers {
|
||||
registry.UserAccessPolicies[userID] = portainer.AccessPolicy{}
|
||||
}
|
||||
|
||||
registry.TeamAccessPolicies = make(portainer.TeamAccessPolicies)
|
||||
for _, teamID := range registry.AuthorizedTeams {
|
||||
registry.TeamAccessPolicies[teamID] = portainer.AccessPolicy{}
|
||||
}
|
||||
|
||||
err = m.registryService.UpdateRegistry(registry.ID, ®istry)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
16
api/datastore/migrator/migrate_dbversion18.go
Normal file
16
api/datastore/migrator/migrate_dbversion18.go
Normal file
|
|
@ -0,0 +1,16 @@
|
|||
package migrator
|
||||
|
||||
import portainer "github.com/portainer/portainer/api"
|
||||
|
||||
func (m *Migrator) updateSettingsToDBVersion19() error {
|
||||
legacySettings, err := m.settingsService.Settings()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if legacySettings.EdgeAgentCheckinInterval == 0 {
|
||||
legacySettings.EdgeAgentCheckinInterval = portainer.DefaultEdgeAgentCheckinIntervalInSeconds
|
||||
}
|
||||
|
||||
return m.settingsService.UpdateSettings(legacySettings)
|
||||
}
|
||||
57
api/datastore/migrator/migrate_dbversion19.go
Normal file
57
api/datastore/migrator/migrate_dbversion19.go
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
package migrator
|
||||
|
||||
import (
|
||||
"strings"
|
||||
)
|
||||
|
||||
const scheduleScriptExecutionJobType = 1
|
||||
|
||||
func (m *Migrator) updateUsersToDBVersion20() error {
|
||||
return m.authorizationService.UpdateUsersAuthorizations()
|
||||
}
|
||||
|
||||
func (m *Migrator) updateSettingsToDBVersion20() error {
|
||||
legacySettings, err := m.settingsService.Settings()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
legacySettings.AllowVolumeBrowserForRegularUsers = false
|
||||
|
||||
return m.settingsService.UpdateSettings(legacySettings)
|
||||
}
|
||||
|
||||
func (m *Migrator) updateSchedulesToDBVersion20() error {
|
||||
legacySchedules, err := m.scheduleService.Schedules()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, schedule := range legacySchedules {
|
||||
if schedule.JobType == scheduleScriptExecutionJobType {
|
||||
if schedule.CronExpression == "0 0 * * *" {
|
||||
schedule.CronExpression = "0 * * * *"
|
||||
} else if schedule.CronExpression == "0 0 0/2 * *" {
|
||||
schedule.CronExpression = "0 */2 * * *"
|
||||
} else if schedule.CronExpression == "0 0 0 * *" {
|
||||
schedule.CronExpression = "0 0 * * *"
|
||||
} else {
|
||||
revisedCronExpression := strings.Split(schedule.CronExpression, " ")
|
||||
if len(revisedCronExpression) == 5 {
|
||||
continue
|
||||
}
|
||||
|
||||
revisedCronExpression = revisedCronExpression[1:]
|
||||
schedule.CronExpression = strings.Join(revisedCronExpression, " ")
|
||||
}
|
||||
|
||||
err := m.scheduleService.UpdateSchedule(schedule.ID, &schedule)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
85
api/datastore/migrator/migrate_dbversion20.go
Normal file
85
api/datastore/migrator/migrate_dbversion20.go
Normal file
|
|
@ -0,0 +1,85 @@
|
|||
package migrator
|
||||
|
||||
import (
|
||||
portainer "github.com/portainer/portainer/api"
|
||||
"github.com/portainer/portainer/api/internal/authorization"
|
||||
)
|
||||
|
||||
func (m *Migrator) updateResourceControlsToDBVersion22() error {
|
||||
legacyResourceControls, err := m.resourceControlService.ResourceControls()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, resourceControl := range legacyResourceControls {
|
||||
resourceControl.AdministratorsOnly = false
|
||||
|
||||
err := m.resourceControlService.UpdateResourceControl(resourceControl.ID, &resourceControl)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *Migrator) updateUsersAndRolesToDBVersion22() error {
|
||||
legacyUsers, err := m.userService.Users()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
settings, err := m.settingsService.Settings()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, user := range legacyUsers {
|
||||
user.PortainerAuthorizations = authorization.DefaultPortainerAuthorizations()
|
||||
err = m.userService.UpdateUser(user.ID, &user)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
endpointAdministratorRole, err := m.roleService.Role(portainer.RoleID(1))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
endpointAdministratorRole.Priority = 1
|
||||
endpointAdministratorRole.Authorizations = authorization.DefaultEndpointAuthorizationsForEndpointAdministratorRole()
|
||||
|
||||
err = m.roleService.UpdateRole(endpointAdministratorRole.ID, endpointAdministratorRole)
|
||||
|
||||
helpDeskRole, err := m.roleService.Role(portainer.RoleID(2))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
helpDeskRole.Priority = 2
|
||||
helpDeskRole.Authorizations = authorization.DefaultEndpointAuthorizationsForHelpDeskRole(settings.AllowVolumeBrowserForRegularUsers)
|
||||
|
||||
err = m.roleService.UpdateRole(helpDeskRole.ID, helpDeskRole)
|
||||
|
||||
standardUserRole, err := m.roleService.Role(portainer.RoleID(3))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
standardUserRole.Priority = 3
|
||||
standardUserRole.Authorizations = authorization.DefaultEndpointAuthorizationsForStandardUserRole(settings.AllowVolumeBrowserForRegularUsers)
|
||||
|
||||
err = m.roleService.UpdateRole(standardUserRole.ID, standardUserRole)
|
||||
|
||||
readOnlyUserRole, err := m.roleService.Role(portainer.RoleID(4))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
readOnlyUserRole.Priority = 4
|
||||
readOnlyUserRole.Authorizations = authorization.DefaultEndpointAuthorizationsForReadOnlyUserRole(settings.AllowVolumeBrowserForRegularUsers)
|
||||
|
||||
err = m.roleService.UpdateRole(readOnlyUserRole.ID, readOnlyUserRole)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return m.authorizationService.UpdateUsersAuthorizations()
|
||||
}
|
||||
94
api/datastore/migrator/migrate_dbversion22.go
Normal file
94
api/datastore/migrator/migrate_dbversion22.go
Normal file
|
|
@ -0,0 +1,94 @@
|
|||
package migrator
|
||||
|
||||
import portainer "github.com/portainer/portainer/api"
|
||||
|
||||
func (m *Migrator) updateTagsToDBVersion23() error {
|
||||
migrateLog.Info("Updating tags")
|
||||
tags, err := m.tagService.Tags()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, tag := range tags {
|
||||
tag.EndpointGroups = make(map[portainer.EndpointGroupID]bool)
|
||||
tag.Endpoints = make(map[portainer.EndpointID]bool)
|
||||
err = m.tagService.UpdateTag(tag.ID, &tag)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *Migrator) updateEndpointsAndEndpointGroupsToDBVersion23() error {
|
||||
migrateLog.Info("Updating endpoints and endpoint groups")
|
||||
tags, err := m.tagService.Tags()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
tagsNameMap := make(map[string]portainer.Tag)
|
||||
for _, tag := range tags {
|
||||
tagsNameMap[tag.Name] = tag
|
||||
}
|
||||
|
||||
endpoints, err := m.endpointService.Endpoints()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, endpoint := range endpoints {
|
||||
endpointTags := make([]portainer.TagID, 0)
|
||||
for _, tagName := range endpoint.Tags {
|
||||
tag, ok := tagsNameMap[tagName]
|
||||
if ok {
|
||||
endpointTags = append(endpointTags, tag.ID)
|
||||
tag.Endpoints[endpoint.ID] = true
|
||||
}
|
||||
}
|
||||
endpoint.TagIDs = endpointTags
|
||||
err = m.endpointService.UpdateEndpoint(endpoint.ID, &endpoint)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
relation := &portainer.EndpointRelation{
|
||||
EndpointID: endpoint.ID,
|
||||
EdgeStacks: map[portainer.EdgeStackID]bool{},
|
||||
}
|
||||
|
||||
err = m.endpointRelationService.Create(relation)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
endpointGroups, err := m.endpointGroupService.EndpointGroups()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, endpointGroup := range endpointGroups {
|
||||
endpointGroupTags := make([]portainer.TagID, 0)
|
||||
for _, tagName := range endpointGroup.Tags {
|
||||
tag, ok := tagsNameMap[tagName]
|
||||
if ok {
|
||||
endpointGroupTags = append(endpointGroupTags, tag.ID)
|
||||
tag.EndpointGroups[endpointGroup.ID] = true
|
||||
}
|
||||
}
|
||||
endpointGroup.TagIDs = endpointGroupTags
|
||||
err = m.endpointGroupService.UpdateEndpointGroup(endpointGroup.ID, &endpointGroup)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
for _, tag := range tagsNameMap {
|
||||
err = m.tagService.UpdateTag(tag.ID, &tag)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
37
api/datastore/migrator/migrate_dbversion23.go
Normal file
37
api/datastore/migrator/migrate_dbversion23.go
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
package migrator
|
||||
|
||||
import portainer "github.com/portainer/portainer/api"
|
||||
|
||||
func (m *Migrator) updateSettingsToDB24() error {
|
||||
migrateLog.Info("Updating Settings")
|
||||
|
||||
legacySettings, err := m.settingsService.Settings()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
legacySettings.AllowHostNamespaceForRegularUsers = true
|
||||
legacySettings.AllowDeviceMappingForRegularUsers = true
|
||||
legacySettings.AllowStackManagementForRegularUsers = true
|
||||
|
||||
return m.settingsService.UpdateSettings(legacySettings)
|
||||
}
|
||||
|
||||
func (m *Migrator) updateStacksToDB24() error {
|
||||
migrateLog.Info("Updating stacks")
|
||||
stacks, err := m.stackService.Stacks()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for idx := range stacks {
|
||||
stack := &stacks[idx]
|
||||
stack.Status = portainer.StackStatusActive
|
||||
err := m.stackService.UpdateStack(stack.ID, stack)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
25
api/datastore/migrator/migrate_dbversion24.go
Normal file
25
api/datastore/migrator/migrate_dbversion24.go
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
package migrator
|
||||
|
||||
import (
|
||||
portainer "github.com/portainer/portainer/api"
|
||||
)
|
||||
|
||||
func (m *Migrator) updateSettingsToDB25() error {
|
||||
migrateLog.Info("Updating settings")
|
||||
|
||||
legacySettings, err := m.settingsService.Settings()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if legacySettings.TemplatesURL == "" {
|
||||
legacySettings.TemplatesURL = portainer.DefaultTemplatesURL
|
||||
}
|
||||
|
||||
legacySettings.UserSessionTimeout = portainer.DefaultUserSessionTimeout
|
||||
legacySettings.EnableTelemetry = true
|
||||
|
||||
legacySettings.AllowContainerCapabilitiesForRegularUsers = true
|
||||
|
||||
return m.settingsService.UpdateSettings(legacySettings)
|
||||
}
|
||||
52
api/datastore/migrator/migrate_dbversion25.go
Normal file
52
api/datastore/migrator/migrate_dbversion25.go
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
package migrator
|
||||
|
||||
import (
|
||||
portainer "github.com/portainer/portainer/api"
|
||||
)
|
||||
|
||||
func (m *Migrator) updateEndpointSettingsToDB25() error {
|
||||
migrateLog.Info("Updating endpoint settings")
|
||||
settings, err := m.settingsService.Settings()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
endpoints, err := m.endpointService.Endpoints()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for i := range endpoints {
|
||||
endpoint := endpoints[i]
|
||||
|
||||
securitySettings := portainer.EndpointSecuritySettings{}
|
||||
|
||||
if endpoint.Type == portainer.EdgeAgentOnDockerEnvironment ||
|
||||
endpoint.Type == portainer.AgentOnDockerEnvironment ||
|
||||
endpoint.Type == portainer.DockerEnvironment {
|
||||
|
||||
securitySettings = portainer.EndpointSecuritySettings{
|
||||
AllowBindMountsForRegularUsers: settings.AllowBindMountsForRegularUsers,
|
||||
AllowContainerCapabilitiesForRegularUsers: settings.AllowContainerCapabilitiesForRegularUsers,
|
||||
AllowDeviceMappingForRegularUsers: settings.AllowDeviceMappingForRegularUsers,
|
||||
AllowHostNamespaceForRegularUsers: settings.AllowHostNamespaceForRegularUsers,
|
||||
AllowPrivilegedModeForRegularUsers: settings.AllowPrivilegedModeForRegularUsers,
|
||||
AllowStackManagementForRegularUsers: settings.AllowStackManagementForRegularUsers,
|
||||
}
|
||||
|
||||
if endpoint.Type == portainer.AgentOnDockerEnvironment || endpoint.Type == portainer.EdgeAgentOnDockerEnvironment {
|
||||
securitySettings.AllowVolumeBrowserForRegularUsers = settings.AllowVolumeBrowserForRegularUsers
|
||||
securitySettings.EnableHostManagementFeatures = settings.EnableHostManagementFeatures
|
||||
}
|
||||
}
|
||||
|
||||
endpoint.SecuritySettings = securitySettings
|
||||
|
||||
err = m.endpointService.UpdateEndpoint(endpoint.ID, &endpoint)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
41
api/datastore/migrator/migrate_dbversion26.go
Normal file
41
api/datastore/migrator/migrate_dbversion26.go
Normal file
|
|
@ -0,0 +1,41 @@
|
|||
package migrator
|
||||
|
||||
import (
|
||||
portainer "github.com/portainer/portainer/api"
|
||||
"github.com/portainer/portainer/api/dataservices/errors"
|
||||
"github.com/portainer/portainer/api/internal/stackutils"
|
||||
)
|
||||
|
||||
func (m *Migrator) updateStackResourceControlToDB27() error {
|
||||
migrateLog.Info("Updating stack resource controls")
|
||||
resourceControls, err := m.resourceControlService.ResourceControls()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, resource := range resourceControls {
|
||||
if resource.Type != portainer.StackResourceControl {
|
||||
continue
|
||||
}
|
||||
|
||||
stackName := resource.ResourceID
|
||||
|
||||
stack, err := m.stackService.StackByName(stackName)
|
||||
if err != nil {
|
||||
if err == errors.ErrObjectNotFound {
|
||||
continue
|
||||
}
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
resource.ResourceID = stackutils.ResourceControlID(stack.EndpointID, stack.Name)
|
||||
|
||||
err = m.resourceControlService.UpdateResourceControl(resource.ID, &resource)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
24
api/datastore/migrator/migrate_dbversion29.go
Normal file
24
api/datastore/migrator/migrate_dbversion29.go
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
package migrator
|
||||
|
||||
func (m *Migrator) migrateDBVersionToDB30() error {
|
||||
migrateLog.Info("Updating legacy settings")
|
||||
if err := m.MigrateSettingsToDB30(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// so setting to false and "", is what would happen without this code
|
||||
// I'm going to bet there's zero point to changing the value inthe DB
|
||||
// Public for testing
|
||||
func (m *Migrator) MigrateSettingsToDB30() error {
|
||||
legacySettings, err := m.settingsService.Settings()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
legacySettings.OAuthSettings.SSO = false
|
||||
legacySettings.OAuthSettings.LogoutURI = ""
|
||||
return m.settingsService.UpdateSettings(legacySettings)
|
||||
}
|
||||
282
api/datastore/migrator/migrate_dbversion31.go
Normal file
282
api/datastore/migrator/migrate_dbversion31.go
Normal file
|
|
@ -0,0 +1,282 @@
|
|||
package migrator
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"github.com/portainer/portainer/api/dataservices/errors"
|
||||
"log"
|
||||
|
||||
portainer "github.com/portainer/portainer/api"
|
||||
"github.com/portainer/portainer/api/internal/endpointutils"
|
||||
snapshotutils "github.com/portainer/portainer/api/internal/snapshot"
|
||||
)
|
||||
|
||||
func (m *Migrator) migrateDBVersionToDB32() error {
|
||||
migrateLog.Info("Updating registries")
|
||||
err := m.updateRegistriesToDB32()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
migrateLog.Info("Updating dockerhub")
|
||||
err = m.updateDockerhubToDB32()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
migrateLog.Info("Updating resource controls")
|
||||
if err := m.updateVolumeResourceControlToDB32(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
migrateLog.Info("Updating kubeconfig expiry")
|
||||
if err := m.kubeconfigExpiryToDB32(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
migrateLog.Info("Setting default helm repository url")
|
||||
if err := m.helmRepositoryURLToDB32(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *Migrator) updateRegistriesToDB32() error {
|
||||
registries, err := m.registryService.Registries()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
endpoints, err := m.endpointService.Endpoints()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, registry := range registries {
|
||||
|
||||
registry.RegistryAccesses = portainer.RegistryAccesses{}
|
||||
|
||||
for _, endpoint := range endpoints {
|
||||
|
||||
filteredUserAccessPolicies := portainer.UserAccessPolicies{}
|
||||
for userId, registryPolicy := range registry.UserAccessPolicies {
|
||||
if _, found := endpoint.UserAccessPolicies[userId]; found {
|
||||
filteredUserAccessPolicies[userId] = registryPolicy
|
||||
}
|
||||
}
|
||||
|
||||
filteredTeamAccessPolicies := portainer.TeamAccessPolicies{}
|
||||
for teamId, registryPolicy := range registry.TeamAccessPolicies {
|
||||
if _, found := endpoint.TeamAccessPolicies[teamId]; found {
|
||||
filteredTeamAccessPolicies[teamId] = registryPolicy
|
||||
}
|
||||
}
|
||||
|
||||
registry.RegistryAccesses[endpoint.ID] = portainer.RegistryAccessPolicies{
|
||||
UserAccessPolicies: filteredUserAccessPolicies,
|
||||
TeamAccessPolicies: filteredTeamAccessPolicies,
|
||||
Namespaces: []string{},
|
||||
}
|
||||
}
|
||||
m.registryService.UpdateRegistry(registry.ID, ®istry)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *Migrator) updateDockerhubToDB32() error {
|
||||
dockerhub, err := m.dockerhubService.DockerHub()
|
||||
if err == errors.ErrObjectNotFound {
|
||||
return nil
|
||||
} else if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if !dockerhub.Authentication {
|
||||
return nil
|
||||
}
|
||||
|
||||
registry := &portainer.Registry{
|
||||
Type: portainer.DockerHubRegistry,
|
||||
Name: "Dockerhub (authenticated - migrated)",
|
||||
URL: "docker.io",
|
||||
Authentication: true,
|
||||
Username: dockerhub.Username,
|
||||
Password: dockerhub.Password,
|
||||
RegistryAccesses: portainer.RegistryAccesses{},
|
||||
}
|
||||
|
||||
// The following code will make this function idempotent.
|
||||
// i.e. if run again, it will not change the data. It will ensure that
|
||||
// we only have one migrated registry entry. Duplicates will be removed
|
||||
// if they exist and which has been happening due to earlier migration bugs
|
||||
migrated := false
|
||||
registries, _ := m.registryService.Registries()
|
||||
for _, r := range registries {
|
||||
if r.Type == registry.Type &&
|
||||
r.Name == registry.Name &&
|
||||
r.URL == registry.URL &&
|
||||
r.Authentication == registry.Authentication {
|
||||
|
||||
if !migrated {
|
||||
// keep this one entry
|
||||
migrated = true
|
||||
} else {
|
||||
// delete subsequent duplicates
|
||||
m.registryService.DeleteRegistry(portainer.RegistryID(r.ID))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if migrated {
|
||||
return nil
|
||||
}
|
||||
|
||||
endpoints, err := m.endpointService.Endpoints()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for _, endpoint := range endpoints {
|
||||
|
||||
if endpoint.Type != portainer.KubernetesLocalEnvironment &&
|
||||
endpoint.Type != portainer.AgentOnKubernetesEnvironment &&
|
||||
endpoint.Type != portainer.EdgeAgentOnKubernetesEnvironment {
|
||||
|
||||
userAccessPolicies := portainer.UserAccessPolicies{}
|
||||
for userId := range endpoint.UserAccessPolicies {
|
||||
if _, found := endpoint.UserAccessPolicies[userId]; found {
|
||||
userAccessPolicies[userId] = portainer.AccessPolicy{
|
||||
RoleID: 0,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
teamAccessPolicies := portainer.TeamAccessPolicies{}
|
||||
for teamId := range endpoint.TeamAccessPolicies {
|
||||
if _, found := endpoint.TeamAccessPolicies[teamId]; found {
|
||||
teamAccessPolicies[teamId] = portainer.AccessPolicy{
|
||||
RoleID: 0,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
registry.RegistryAccesses[endpoint.ID] = portainer.RegistryAccessPolicies{
|
||||
UserAccessPolicies: userAccessPolicies,
|
||||
TeamAccessPolicies: teamAccessPolicies,
|
||||
Namespaces: []string{},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return m.registryService.Create(registry)
|
||||
}
|
||||
|
||||
func (m *Migrator) updateVolumeResourceControlToDB32() error {
|
||||
endpoints, err := m.endpointService.Endpoints()
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed fetching environments: %w", err)
|
||||
}
|
||||
|
||||
resourceControls, err := m.resourceControlService.ResourceControls()
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed fetching resource controls: %w", err)
|
||||
}
|
||||
|
||||
toUpdate := map[portainer.ResourceControlID]string{}
|
||||
volumeResourceControls := map[string]*portainer.ResourceControl{}
|
||||
|
||||
for i := range resourceControls {
|
||||
resourceControl := resourceControls[i]
|
||||
if resourceControl.Type == portainer.VolumeResourceControl {
|
||||
volumeResourceControls[resourceControl.ResourceID] = &resourceControl
|
||||
}
|
||||
}
|
||||
|
||||
for _, endpoint := range endpoints {
|
||||
if !endpointutils.IsDockerEndpoint(&endpoint) {
|
||||
continue
|
||||
}
|
||||
|
||||
totalSnapshots := len(endpoint.Snapshots)
|
||||
if totalSnapshots == 0 {
|
||||
log.Println("[DEBUG] [volume migration] [message: no snapshot found]")
|
||||
continue
|
||||
}
|
||||
|
||||
snapshot := endpoint.Snapshots[totalSnapshots-1]
|
||||
|
||||
endpointDockerID, err := snapshotutils.FetchDockerID(snapshot)
|
||||
if err != nil {
|
||||
log.Printf("[WARN] [database,migrator,v31] [message: failed fetching environment docker id] [err: %s]", err)
|
||||
continue
|
||||
}
|
||||
|
||||
if volumesData, done := snapshot.SnapshotRaw.Volumes.(map[string]interface{}); done {
|
||||
if volumesData["Volumes"] == nil {
|
||||
log.Println("[DEBUG] [volume migration] [message: no volume data found]")
|
||||
continue
|
||||
}
|
||||
|
||||
findResourcesToUpdateForDB32(endpointDockerID, volumesData, toUpdate, volumeResourceControls)
|
||||
}
|
||||
}
|
||||
|
||||
for _, resourceControl := range volumeResourceControls {
|
||||
if newResourceID, ok := toUpdate[resourceControl.ID]; ok {
|
||||
resourceControl.ResourceID = newResourceID
|
||||
err := m.resourceControlService.UpdateResourceControl(resourceControl.ID, resourceControl)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed updating resource control %d: %w", resourceControl.ID, err)
|
||||
}
|
||||
|
||||
} else {
|
||||
err := m.resourceControlService.DeleteResourceControl(resourceControl.ID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed deleting resource control %d: %w", resourceControl.ID, err)
|
||||
}
|
||||
log.Printf("[DEBUG] [volume migration] [message: legacy resource control(%s) has been deleted]", resourceControl.ResourceID)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func findResourcesToUpdateForDB32(dockerID string, volumesData map[string]interface{}, toUpdate map[portainer.ResourceControlID]string, volumeResourceControls map[string]*portainer.ResourceControl) {
|
||||
volumes := volumesData["Volumes"].([]interface{})
|
||||
for _, volumeMeta := range volumes {
|
||||
volume := volumeMeta.(map[string]interface{})
|
||||
volumeName, nameExist := volume["Name"].(string)
|
||||
if !nameExist {
|
||||
continue
|
||||
}
|
||||
createTime, createTimeExist := volume["CreatedAt"].(string)
|
||||
if !createTimeExist {
|
||||
continue
|
||||
}
|
||||
|
||||
oldResourceID := fmt.Sprintf("%s%s", volumeName, createTime)
|
||||
resourceControl, ok := volumeResourceControls[oldResourceID]
|
||||
|
||||
if ok {
|
||||
toUpdate[resourceControl.ID] = fmt.Sprintf("%s_%s", volumeName, dockerID)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (m *Migrator) kubeconfigExpiryToDB32() error {
|
||||
settings, err := m.settingsService.Settings()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
settings.KubeconfigExpiry = portainer.DefaultKubeconfigExpiry
|
||||
return m.settingsService.UpdateSettings(settings)
|
||||
}
|
||||
|
||||
func (m *Migrator) helmRepositoryURLToDB32() error {
|
||||
settings, err := m.settingsService.Settings()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
settings.HelmRepositoryURL = portainer.DefaultHelmRepositoryURL
|
||||
return m.settingsService.UpdateSettings(settings)
|
||||
}
|
||||
22
api/datastore/migrator/migrate_dbversion32.go
Normal file
22
api/datastore/migrator/migrate_dbversion32.go
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
package migrator
|
||||
|
||||
import portainer "github.com/portainer/portainer/api"
|
||||
|
||||
func (m *Migrator) migrateDBVersionToDB33() error {
|
||||
if err := m.migrateSettingsToDB33(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *Migrator) migrateSettingsToDB33() error {
|
||||
settings, err := m.settingsService.Settings()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
migrateLog.Info("Setting default kubectl shell image")
|
||||
settings.KubectlShellImage = portainer.DefaultKubectlShellImage
|
||||
return m.settingsService.UpdateSettings(settings)
|
||||
}
|
||||
32
api/datastore/migrator/migrate_dbversion33.go
Normal file
32
api/datastore/migrator/migrate_dbversion33.go
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
package migrator
|
||||
|
||||
import "github.com/portainer/portainer/api/dataservices"
|
||||
|
||||
func (m *Migrator) migrateDBVersionToDB34() error {
|
||||
migrateLog.Info("Migrating stacks")
|
||||
err := MigrateStackEntryPoint(m.stackService)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// MigrateStackEntryPoint exported for testing (blah.)
|
||||
func MigrateStackEntryPoint(stackService dataservices.StackService) error {
|
||||
stacks, err := stackService.Stacks()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for i := range stacks {
|
||||
stack := &stacks[i]
|
||||
if stack.GitConfig == nil {
|
||||
continue
|
||||
}
|
||||
stack.GitConfig.ConfigFilePath = stack.EntryPoint
|
||||
if err := stackService.UpdateStack(stack.ID, stack); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
12
api/datastore/migrator/migrate_dbversion34.go
Normal file
12
api/datastore/migrator/migrate_dbversion34.go
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
package migrator
|
||||
|
||||
func (m *Migrator) migrateDBVersionToDB35() error {
|
||||
// These should have been migrated already, but due to an earlier bug and a bunch of duplicates,
|
||||
// calling it again will now fix the issue as the function has been repaired.
|
||||
migrateLog.Info("Updating dockerhub registries")
|
||||
err := m.updateDockerhubToDB32()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
94
api/datastore/migrator/migrate_dbversion34_test.go
Normal file
94
api/datastore/migrator/migrate_dbversion34_test.go
Normal file
|
|
@ -0,0 +1,94 @@
|
|||
package migrator
|
||||
|
||||
const (
|
||||
db35TestFile = "portainer-mig-35.db"
|
||||
username = "portainer"
|
||||
password = "password"
|
||||
)
|
||||
|
||||
// TODO: this is exactly the kind of reaching into the internals of the store we should not do
|
||||
// func setupDB35Test(t *testing.T) *Migrator {
|
||||
// is := assert.New(t)
|
||||
// dbConn, err := bolt.Open(path.Join(t.TempDir(), db35TestFile), 0600, &bolt.Options{Timeout: 1 * time.Second})
|
||||
// is.NoError(err, "failed to init testing DB connection")
|
||||
|
||||
// // Create an old style dockerhub authenticated account
|
||||
// dockerhubService, err := dockerhub.NewService(&database.DbConnection{DB: dbConn})
|
||||
// is.NoError(err, "failed to init testing registry service")
|
||||
// err = dockerhubService.UpdateDockerHub(&portainer.DockerHub{true, username, password})
|
||||
// is.NoError(err, "failed to create dockerhub account")
|
||||
|
||||
// registryService, err := registry.NewService(&database.DbConnection{DB: dbConn})
|
||||
// is.NoError(err, "failed to init testing registry service")
|
||||
|
||||
// endpointService, err := endpoint.NewService(&database.DbConnection{DB: dbConn})
|
||||
// is.NoError(err, "failed to init endpoint service")
|
||||
|
||||
// m := &Migrator{
|
||||
// db: dbConn,
|
||||
// dockerhubService: dockerhubService,
|
||||
// registryService: registryService,
|
||||
// endpointService: endpointService,
|
||||
// }
|
||||
|
||||
// return m
|
||||
// }
|
||||
|
||||
// // TestUpdateDockerhubToDB32 tests a normal upgrade
|
||||
// func TestUpdateDockerhubToDB32(t *testing.T) {
|
||||
// is := assert.New(t)
|
||||
// m := setupDB35Test(t)
|
||||
// defer m.db.Close()
|
||||
// defer os.Remove(db35TestFile)
|
||||
|
||||
// if err := m.updateDockerhubToDB32(); err != nil {
|
||||
// t.Errorf("failed to update settings: %v", err)
|
||||
// }
|
||||
|
||||
// // Verify we have a single registry were created
|
||||
// registries, err := m.registryService.Registries()
|
||||
// is.NoError(err, "failed to read registries from the RegistryService")
|
||||
// is.Equal(len(registries), 1, "only one migrated registry expected")
|
||||
// }
|
||||
|
||||
// // TestUpdateDockerhubToDB32_with_duplicate_migrations tests an upgrade where in earlier versions a broken migration
|
||||
// // created a large number of duplicate "dockerhub migrated" registry entries.
|
||||
// func TestUpdateDockerhubToDB32_with_duplicate_migrations(t *testing.T) {
|
||||
// is := assert.New(t)
|
||||
// m := setupDB35Test(t)
|
||||
// defer m.db.Close()
|
||||
// defer os.Remove(db35TestFile)
|
||||
|
||||
// // Create lots of duplicate entries...
|
||||
// registry := &portainer.Registry{
|
||||
// Type: portainer.DockerHubRegistry,
|
||||
// Name: "Dockerhub (authenticated - migrated)",
|
||||
// URL: "docker.io",
|
||||
// Authentication: true,
|
||||
// Username: "portainer",
|
||||
// Password: "password",
|
||||
// RegistryAccesses: portainer.RegistryAccesses{},
|
||||
// }
|
||||
|
||||
// for i := 1; i < 150; i++ {
|
||||
// err := m.registryService.CreateRegistry(registry)
|
||||
// assert.NoError(t, err, "create registry failed")
|
||||
// }
|
||||
|
||||
// // Verify they were created
|
||||
// registries, err := m.registryService.Registries()
|
||||
// is.NoError(err, "failed to read registries from the RegistryService")
|
||||
// is.Condition(func() bool {
|
||||
// return len(registries) > 1
|
||||
// }, "expected multiple duplicate registry entries")
|
||||
|
||||
// // Now run the migrator
|
||||
// if err := m.updateDockerhubToDB32(); err != nil {
|
||||
// t.Errorf("failed to update settings: %v", err)
|
||||
// }
|
||||
|
||||
// // Verify we have a single registry were created
|
||||
// registries, err = m.registryService.Registries()
|
||||
// is.NoError(err, "failed to read registries from the RegistryService")
|
||||
// is.Equal(len(registries), 1, "only one migrated registry expected")
|
||||
// }
|
||||
100
api/datastore/migrator/migrator.go
Normal file
100
api/datastore/migrator/migrator.go
Normal file
|
|
@ -0,0 +1,100 @@
|
|||
package migrator
|
||||
|
||||
import (
|
||||
portainer "github.com/portainer/portainer/api"
|
||||
"github.com/portainer/portainer/api/dataservices/dockerhub"
|
||||
"github.com/portainer/portainer/api/dataservices/endpoint"
|
||||
"github.com/portainer/portainer/api/dataservices/endpointgroup"
|
||||
"github.com/portainer/portainer/api/dataservices/endpointrelation"
|
||||
"github.com/portainer/portainer/api/dataservices/extension"
|
||||
"github.com/portainer/portainer/api/dataservices/registry"
|
||||
"github.com/portainer/portainer/api/dataservices/resourcecontrol"
|
||||
"github.com/portainer/portainer/api/dataservices/role"
|
||||
"github.com/portainer/portainer/api/dataservices/schedule"
|
||||
"github.com/portainer/portainer/api/dataservices/settings"
|
||||
"github.com/portainer/portainer/api/dataservices/stack"
|
||||
"github.com/portainer/portainer/api/dataservices/tag"
|
||||
"github.com/portainer/portainer/api/dataservices/teammembership"
|
||||
"github.com/portainer/portainer/api/dataservices/user"
|
||||
"github.com/portainer/portainer/api/dataservices/version"
|
||||
plog "github.com/portainer/portainer/api/datastore/log"
|
||||
"github.com/portainer/portainer/api/internal/authorization"
|
||||
)
|
||||
|
||||
var migrateLog = plog.NewScopedLog("database, migrate")
|
||||
|
||||
type (
|
||||
// Migrator defines a service to migrate data after a Portainer version update.
|
||||
Migrator struct {
|
||||
currentDBVersion int
|
||||
|
||||
endpointGroupService *endpointgroup.Service
|
||||
endpointService *endpoint.Service
|
||||
endpointRelationService *endpointrelation.Service
|
||||
extensionService *extension.Service
|
||||
registryService *registry.Service
|
||||
resourceControlService *resourcecontrol.Service
|
||||
roleService *role.Service
|
||||
scheduleService *schedule.Service
|
||||
settingsService *settings.Service
|
||||
stackService *stack.Service
|
||||
tagService *tag.Service
|
||||
teamMembershipService *teammembership.Service
|
||||
userService *user.Service
|
||||
versionService *version.Service
|
||||
fileService portainer.FileService
|
||||
authorizationService *authorization.Service
|
||||
dockerhubService *dockerhub.Service
|
||||
}
|
||||
|
||||
// MigratorParameters represents the required parameters to create a new Migrator instance.
|
||||
MigratorParameters struct {
|
||||
DatabaseVersion int
|
||||
EndpointGroupService *endpointgroup.Service
|
||||
EndpointService *endpoint.Service
|
||||
EndpointRelationService *endpointrelation.Service
|
||||
ExtensionService *extension.Service
|
||||
RegistryService *registry.Service
|
||||
ResourceControlService *resourcecontrol.Service
|
||||
RoleService *role.Service
|
||||
ScheduleService *schedule.Service
|
||||
SettingsService *settings.Service
|
||||
StackService *stack.Service
|
||||
TagService *tag.Service
|
||||
TeamMembershipService *teammembership.Service
|
||||
UserService *user.Service
|
||||
VersionService *version.Service
|
||||
FileService portainer.FileService
|
||||
AuthorizationService *authorization.Service
|
||||
DockerhubService *dockerhub.Service
|
||||
}
|
||||
)
|
||||
|
||||
// NewMigrator creates a new Migrator.
|
||||
func NewMigrator(parameters *MigratorParameters) *Migrator {
|
||||
return &Migrator{
|
||||
currentDBVersion: parameters.DatabaseVersion,
|
||||
endpointGroupService: parameters.EndpointGroupService,
|
||||
endpointService: parameters.EndpointService,
|
||||
endpointRelationService: parameters.EndpointRelationService,
|
||||
extensionService: parameters.ExtensionService,
|
||||
registryService: parameters.RegistryService,
|
||||
resourceControlService: parameters.ResourceControlService,
|
||||
roleService: parameters.RoleService,
|
||||
scheduleService: parameters.ScheduleService,
|
||||
settingsService: parameters.SettingsService,
|
||||
tagService: parameters.TagService,
|
||||
teamMembershipService: parameters.TeamMembershipService,
|
||||
stackService: parameters.StackService,
|
||||
userService: parameters.UserService,
|
||||
versionService: parameters.VersionService,
|
||||
fileService: parameters.FileService,
|
||||
authorizationService: parameters.AuthorizationService,
|
||||
dockerhubService: parameters.DockerhubService,
|
||||
}
|
||||
}
|
||||
|
||||
// Version exposes version of database
|
||||
func (migrator *Migrator) Version() int {
|
||||
return migrator.currentDBVersion
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue